Augmenting Clinical Intelligence with Federated Learning, Imaging Analytics and Outcomes Decision Support

ABSTRACT

The invention integrates emerging applications, tools and techniques for machine learning in medicine with videoconference networking technology in novel business methods that support rapid adaptive learning for medical minds and machines. These methods can leverage domain knowledge and clinical expertise with networked cognitive collaboration, augmented clinical intelligence and cybernetic workflow streams for learning health care systems. The invention enables multimodal clinical communications, collaboration, consultation and instruction between and among heterogeneous networked teams of persons, machines, devices, neural networks, robots and algorithms. It provides for both synchronous and asynchronous cognitive collaboration with multichannel, multiplexed imagery data streams during various stages of medical disease and injury management—detection, diagnosis, prognosis, treatment, measurement, monitoring and reporting, as well as workflow optimization with operational analytics for outcomes, performance, results, resource utilization, resource consumption and costs. The invention enables cognitively-enriched, annotation and tagging, as well as encapsulation, saving and sharing of collaborated imagery data streams as packetized clinical intelligence.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is Continuation-in-part of U.S. application Ser. No.16/450,974 filed Jun. 24, 2019 entitled:

-   -   “Cognitive Collaboration with Neurosynaptic Imaging Networks,        Augmented Medical Intelligence and Cybernetic Workflow Streams”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application is Continuation-in-part of U.S. application        Ser. No. 15/731,201 filed May 2, 2017 entitled:    -   “Cognitive Collaboration with Neurosynaptic Imaging Networks,        Augmented Medical Intelligence and Cybernetic Workflow Streams”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application is Continuation-in-part of U.S. application        Ser. No. 14/544,807 filed Feb. 18, 2015 entitled:    -   “Multimodal Cognitive Communications and Collaborative Knowledge        Exchange with Visual Neural Networking and Packetized Augmented        Intelligence”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application Claims Priority from U.S. Provisional        Application 61/967,323 filed Mar. 15, 2014 entitled:    -   “Network systems apparatus and method of use adapted for        tele-visual communications and collaboration with streaming        medical imagery and clinical informatics by networked teams of        minds, machines, languages and tools, including recursively        annotating, tagging, encapsulating and saving shared tele-visual        communications, collaborations, imagery and informatics together        as clinical cognitive vismemes in standard known file formats        for interoperable delivery of personalized medicine”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application is Continuation-in-part of U.S. application        Ser. No. 13/999,688 filed Mar. 15, 2014 entitled:    -   “System and method for recursive cognitive enrichment with        collaborative network exchange of multimodal multistream digital        communications across neurosynaptic butterfly networks”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application Claims Priority from U.S. Provisional        Application 61/852,625 filed Mar. 15, 2013 entitled:    -   “Network apparatus system and method of use adapted for viewing        recursively annotating and tagging, saving and retrieving,        consulting and collaborating with semantically searchable        clinical cognitive vismemes, together with encapsulated metadata        and dicomized image-waveforms, over visual neural networks for        early detection, diagnosis, prognosis, treatment, measurement        and monitoring of disease, including delivery of precision        personalized medicine across interconnected knowledge networks”        naming as inventor James Paul Smurro, which is incorporated        herein by reference in its entirety.        This application may be related to the following commonly        assigned and commonly filed U.S. patent applications, each of        which is incorporated herein by reference in its entirety:    -   1. U.S. Pat. No. 8,924,864 B2 entitled “System and method for        collaboratively communicating on images and saving those        communications and images in a standard known format”, naming as        inventors Mariotti et al, issued Dec. 30, 2014.    -   2. U.S. patent application 20140176661 A1 entitled “System and        method for surgical telementoring and training with virtualized        telestration and haptic holograms, including metadata tagging,        encapsulation and saving multi-modal streaming medical imagery        together with multi-dimensional [4-d] virtual mesh and        multi-sensory annotation in standard file formats used for        digital imaging and communications in medicine (dicom)”, naming        as inventors Smurro et al, published Jun. 26, 2014.

FIELD

The invention generally relates to a network system and methods forreceiving and transmitting streaming imagery data, including medicalimages, waveforms, audio and haptic signals, biomedical and clinicaldocuments, both live and asynchronously, and allowing operators toconcurrently curate, annotate, tag, encapsulate and save that imagerydata, together with those annotations and searchable metadata tags insingle file format structures. The invention acquires streaming imagerydata through network-connected imagery-enabled devices, and allows avariety of cognitive collaborants, singly or together, to concurrentlycommunicate, collaborate, consult and instruct, generally by curating,annotating and tagging, telestrating, sketching image overlays onstreaming imagery data, and saving those images together withcollaborated annotations and metadata, as streaming augmentedintelligence for rapid adaptive learning, specialist skills acquisitionand informatics-enriched innovation with multimodal clinical instructionand value chain knowledge exchange.

IMPROVEMENT OVER PRIOR ART

As used herein, “cognitive collaborant” refers to one or more cognitivecollaborators, human or non-human, including persons, machines, devices,neural networks, robots and algorithms, as well as heterogeneousnetworked teams of persons, machines, devices, neural networks, robotsand algorithms. The invention enables multichannel multiplexedcommunications, collaboration, consultation and instruction withstreaming imagery data by cognitive collaborants, includingheterogeneous networked teams of persons, machines, devices, neuralnetworks, robots and algorithms. The invention enables both synchronousand asynchronous multimodal clinical communications, collaboration,consultation and instruction during various stages of medical diseaseand injury management, including detection, diagnosis, prognosis,treatment, measurement, monitoring and reporting, as well as workflowoptimization with operational analytics for outcomes, performance,results, resource utilization, resource consumption and costs.

The invention enables pluribus network encoding with multichannelmultiplexed steaming imagery data from signals, sensors and devices,including packets, waveforms and streams, along with space shifting,time shifting and format shifting media synchronization. The inventionenables heterogeneous networked teams of cognitive collaborants torecursively curate, annotate and tag, encapsulate, save and sharemultichannel multiplexed imagery data streams, including multisensorydata stream visualizations, and bi-directional value chain knowledgeexchange, with streaming imagery data from heterogeneous spatial andtemporal sources, locations, modalities and scales. The invention canacquire both live stream and archived medical imagery data fromnetwork-connected medical devices, cameras, signals and sensors. Thenetwork system can also acquire multiomic data—phenotypic, genomic andmetabolomic, as well as pathomic, radiomic, radiopathomic andradiogenomic—from structured reports and clinical documents, as well asbiometric maps, movies, data stream visualizations, hapmaps and heatmaps. The network system can also acquire packetized clinicalinformatics from imagery data repositories, from clinical workstationsand mobile medical devices, as well as from wearable computing devices,signals and sensors.

The invention enables networked teams to interactively communicate,concurrently collaborate and bi-directionally exchange multichannelmultiplexed imagery data streams, singly or together, in real time orasynchronously, generally by curating, annotating and tagging imageryinformation objects. The invention encapsulates and saves collaboratedimagery data streams, together with collaborated clinical annotations,imaging metadata, as well as semantic metadata and annotations, andprivacy protected metadata identifying personal health information[PHI], in standard known file formats as clinical cognitivevismemes—encapsulated packets, waveforms and streams. Clinical cognitivevismemes preserve packetized imagery information objects, clinicalannotations and metadata tags in native file format structures,including PDF, MPEG, JPEG, XML, XMPP, TIFF, RDF, RDF/XML, QR, SVG andDAE, as well as DICOM. When clinical cognitive vismemes are encapsulatedand saved in formats compliant with standards for digital communicationsin medicine [DICOM], they can also be referred to as medical dicomvismemes.

Clinical cognitive vismemes allow for recursive cognitive enrichmentthrough recursive curation, annotation, tagging, encapsulation andsaving, together with value chain knowledge exchange. Value chainknowledge exchange includes knowledge creation and acquisition,knowledge visualization and sharing, knowledge replication andintegration, knowledge protection and destruction, as well as outcomesperformance evaluation and learning, all of which can accelerateoutcomes-driven innovation.

The invention also enables informatics-enriched innovation and valuechain knowledge exchange with multimodal clinical communications andmultisensory data stream visualization. Multimodal clinicalcommunications, collaboration, consultation and instruction includesmultisensory [sight-sound-touch] digital data exchange with vision,audition and sensation, including semiotics, semantics and somesthetics[haptics].

The invention enables live stream multicasting of N-way multi-partycollaborations, including multisensory data stream visualization andbi-directional knowledge exchange, with multichannel multiplexed imagerydata streams, and concurrent transmission of secure, encrypted clinicalcognitive vismemes across collaborative file sharing data networks forinformatics-enriched learning, specialist skills acquisition andaccelerated knowledge exchange. Principal areas of clinical applicationinclude cognitively-enriched enterprise imaging with streaming imageryinformatics, collaborative precision medicine with multiomic dataanalytics, informatics-enriched imagery guided intervention, includingrobotic-assisted surgery, along with newly-emerging disciplines formachine learning with medical imaging, including deep learning, transferlearning, reinforcement learning, convolutional neural networks,recurrent neural networks, long short term memory networks and naturallanguage processing, along with emerging techniques for precision guidedbiomedical nanorobotics and precision targeted theranostic nanomedicine.

The novelty of the present invention enables multiparty networkedclinical communications, collaboration, consultation and instructionwith streaming imagery data by integrating videoconferencing systemstechnology with emerging applications and techniques for machinelearning in medicine.

In addition to these the present invention supports several additionalimprovements over prior art, which are briefly described below:

1. Federated Learning (FL) is a Machine Learning Setting where ManyClients Collaboratively Train a Model Under the Orchestration of aCentral Server, while Keeping the Training Data Decentralized.

-   -   “Federated learning is a machine learning setting where multiple        entities (clients) collaborate in solving a machine learning        problem, under the coordination of a central server or service        provider. Each client's raw data is stored locally and not        exchanged or transferred; instead, focused updates intended for        immediate aggregation are used to achieve the learning        objective.”        -   Cross-silo applications for electronic health records mining            and medical data segmentation    -   Cross-Silo Federated Learning, Fully Decentralized/Peer-to-Peer        Distributed Learning and Federated transfer learning

Federated learning with edge computing, which affords greater datasecurity, where its data is separately stored and processed in the edgenode, as well as containerization, that provides management of all thevarious execution environments that devices in the edge federatedlearning setting will utilize.

The combination of edge computing and federated learning, along withmore collaborative training methods for edge federated learning, canprovides for better user experience and privacy protection.

2. Integration with Imaging Analytics for Preparing Medical Imaging Datafor Machine Learning

Image analysis is one of the most promising applications of artificialintelligence (AI) in health care, potentially improving prediction,diagnosis, and treatment of diseases. Federated artificial intelligence(AI)-based medical image analysis for the application of AI tolarge-scale clinical imaging data with decentralized local execution ofdata analyses can solve many obstacles of cross-site collaboration.

Supervised artificial intelligence (AI) methods for evaluation ofmedical images require a curation process for data to optimally train,validate, and test algorithms. The chief obstacles to development andclinical implementation of AI algorithms include availability ofsufficiently large, curated, and representative training data thatincludes expert labeling (eg, annotations).

New approaches such as federated learning, interactive reporting, andsynoptic reporting may help to address data availability in the future;however, curating and annotating data, as well as computationalrequirements, are substantial barriers.

Advanced imaging analytics and the extraction of high-dimensional datafrom medical images, called radiomics, is emerging as the other side ofthe personalization coin. By adding an individual patients' tumorphenotypic (structural) information coupled with the patients' geneticdata, drug developers can create more precise therapies.

By using artificial intelligence (AI) to discern and compute data frommedical images, radiomics enables drug developers to profile a patient,tumor, and therapy across multiple dimensions to find patterns andsimilarities that would otherwise be unobtainable.

3. Support for Interactive (“Visually Embedded”) Annotations, as Well asSemantically Meaningful Segmentation, Parametric Maps and StructuredReports

Traditionally, different annotation mechanisms have been provided inDICOM for the purpose of encoding, transporting and querying forclinically generated and machine generated image-related results. Someof these are purely focused on consistent rendering and appearance, andrequire visual human interpretation, while others are structured, codedand semantically meaningful, but require more work on the authoringside.

Modern Al workflow scenarios require more complex semanticallymeaningful payloads and interactions with the users, for training andtesting, as well as for clinical operation. To the extent that DICOMpayloads and protocols can be reused for both types of scenario, theexisting standard should be leveraged to implement new Al applicationsand existing templates and code sets.

The workflow has traditionally focused on human creation and displayrather than automated analysis and consumption. Identify the workflow ofannotation creation and use in Al using DICOM protocols and payloads.

Diagnostic and evidential static image, video clip, and sound multimediaare captured during routine clinical care in cardiology, dermatology,ophthalmology, pathology, physiatry, radiation oncology, radiology,endoscopic procedural specialties, and other medical disciplines.

4. Interactive Multimedia Report (IMR) Creation and Ingestion intoElectronic Health Records.

One consensus definition of IMR is

-   -   “interactive medical documentation that combines clinical        images, videos, sound, imaging metadata, and/or image        annotations with text, typographic emphases, tables, graphs,        event timelines, anatomic maps, hyperlinks, and/or educational        resources to optimize communication between medical        professionals, and between medical professionals and their        patients.”

Providers typically describe the multimedia findings in contemporaneouselectronic health record clinical notes or associate a textualinterpretative report. Visual communication aids commonly used toconnect, synthesize, and supplement multimedia and descriptive textoutside medicine remain technically challenging to integrate intopatient care.

Such beneficial interactive elements may include hyperlinks betweentext, multimedia elements, alphanumeric and geometric annotations,tables, graphs, timelines, diagrams, anatomic maps, and hyperlinks toexternal educational references that patients or provider consumers mayfind valuable.

5. Support for IHE Radiology Newly Emerging Supports for AIInteroperability in Imaging

BACKGROUND Machine Learning in Medicine

Digital clinical data, captured and stored on electronic medical recordsby hospitals and clinics, along with ever growing volumes of medicalimaging data, have sparked growing interest and applications of machinelearning in medicine.

In recent years there has been a proliferation of artificialintelligence (AI) tools and resources available in medicine, especiallywith ever increasing computing power and a growing acceptance of cloudcomputing by hospitals and clinicians. Imaging analysis and clinicaldecision support are two particularly popular applications of machinelearning in medicine, with tools that support diagnosis, treatment, carecoordination and remote monitoring.

There is much promise in the utilization of AI methodologies such asmachine learning and deep learning for augmented biomedical imageinterpretation in radiology, cardiology, pathology, dermatology,ophthalmology and genomic medicine.

One example of machine learning for medical imaging involvesdifferential diagnosis of breast cancer enabled by joint analysis offunctional genomic information and pathology images (pathogenomics)within a biomedical imaging informatics framework consisting of imageextraction, feature combination, and classification.

Algorithms based on deep convolutional neural networks have been used todetect diabetic retinopathy in retinal fundus photographs with highspecificity and sensitivity, as good as with board-certifiedophthalmologists in making diagnoses.

Personalized precision medicine with all its complexity and enormity ofdata to be analyzed is particularly well suited for the portfolio of AImethodologies, including deep learning, which can be used to identifyand assess patients with similar genotype-phenotype characteristics. Ingenomic diagnostics, clinicians are often frustrated by the tediousnature of searching for genotype-phenotype interrelationships amongsyndromes, especially for extremely rare diseases. Now, geneticists maybe able to use visual diagnostic decision support systems that employmachine learning algorithms and digital imaging processing techniques ina hybrid approach for automated detection and diagnosis in medicalgenetics.

An essential part of the precision medicine paradigm is individualizedtherapy based on genotype-phenotype coupling and pharmacogenomicprofiles. There are many potential applications of deep learning forlarge datasets in pharmaceutical research, such as physicochemicalproperty prediction, formulation prediction, and properties such asabsorption, distribution, metabolism, excretion, toxicity, and eventarget prediction.

Surgical robotics have advanced to include 3D visualization andinformatics-enriched imagery guided interventions.

Machine learning algorithms can also be applied to large-scale wearablesensor data in neurological disorders such as Parkinson's disease tosignificantly improve clinical diagnosis and management. Sensor-based,quantitative and objective systems for assessing Parkinson's diseasehave the potential to replace traditional qualitative and subjectiveratings by human interpretation.

An essential part of digital medicine and wearable devices is the datamining of the incoming data for anomaly detection, prediction, diagnosisand clinical decision making. Data mining processes for data streamsfrom wearable devices typically include feature extraction/selectionprocesses to improve detection, prediction, and decision making byclinicians.

Machine learning techniques include supervised methodologies such asneural networks, support vector machines, naïve Bayesian classifiers,and hidden Markov models, as well as semi-supervised methods that can beused with less labeled data. These techniques can be applied tomolecular imaging modalities with promising application for clinicaldiagnosis.

Four types of machine learning—deep learning, reinforcement learning,transfer learning and one-shot learning—may figure prominently in futureapplications of AI in medicine.

Deep learning with all its myriad capabilities may well be used for manyapplications in medical data analytics. The multiple layers of neuralnets can be assigned to the many phenotypic as well as genomicexpressions of conditions such as clinical measurements, biomarkers,imaging data, genomic information and disease subtypes.

Reinforcement learning is ideally designed for the many decision makingaspects of medicine since it readily accommodates recognition of complexpatterns, long-term planning, and many decision-making processes inclinical practice.

Transfer learning occurs when a network that is trained for one task isthen used to configure the network for another task.

One-shot learning can bring a special dimension to unique cases inmedicine as it does not require the usual large dimensionality of datathat the other types of machine learning techniques typically require.

Natural language processing [NLP] includes machine learning techniquesfor speech recognition and identification, as well as languageunderstanding and generation. Medical NLP may become increasingly usefulfor collaborative curation, annotation and tagging of medical imagerydata by heterogeneous teams of medical minds and machines. Curatedmedical images, annotated and tagged as medical “ground truth”, willbecome increasingly important not only for clinical detection, diagnosisand decision support, but also for the training, testing and validationat scale of machine learning algorithms requiring voluminous imagerydata sets. Recurrent Neural Networks [RNN] and Long Short Term Memory[LSTM] networks have been successfully applied to a variety of problemsin speech recognition, language modeling and translation, imagecaptioning and image annotation.

Personalized precision medicine may require disruptive computationalplatforms for new biomedical knowledge discovery, and scalablecomputational frameworks that can leverage hypergraph-based data modelsand query languages that may be well-suited for representing complexmulti-lateral, multi-scalar, and multi-dimensional relationships.Hypergraph-like stores of clinical information (e.g., from diseaseregistries) can be combined with an individual patient's genomic andother phenotypic information (such as imaging data) to create moreprecise and personalized genome-based knowledge stores for clinicaltranslation and discovery. Patients of very similar genomic and clinicalelements could then be better discovered and matched for diagnostic andtherapeutic strategies.

Cloud computing and storage can facilitate a full range of AI techniquesfor multi-institutional collaborations that may become essential todriving future applications of AI in biomedicine and healthcare. Theinternet of medical things (IoMT) may also provide the critical datasources for medicine in the form of wearable and monitoring devices fromboth hospital and home.

Clinical data analytics will increasingly rely on machine learning toolsand techniques to answer many clinical questions for intelligence-basedmedicine, rather than current best practices of principally relying uponpublished medical reports for evidence-based medicine.

There is a compelling need for informatics-enriched innovation withAI-powered technologies that can improve diagnostics and therapeutics,and help deliver value-based care. The convergence of “big data” stores,improved AI algorithms, increasing use of graphical processingcomputational power (GPU), and cloud storage has begun to produce someintriguing machine learning projects with promising results forbiomedicine and healthcare. Perhaps more importantly, continuingadvances with AI-powered tools and techniques in healthcare will requireefforts to ensure more collaborative teamwork and better sharing ofcurated datasets among the various stakeholders.

Productive AI strategies may involve synergistic collaborations ofhumans and machines—clinicians and data scientists, empowered with AI—sothat machine learning in medicine may become a key enabler of newclinical knowledge and augmented clinical intelligence for learninghealth care systems.

Collaborative Clinical Workflows with Enterprise Imaging

The HIMSS-SIIM Collaborative Workgroup has defined Enterprise Imagingas:

-   -   “The management of all clinically relevant content, including        imaging and multimedia, for the purposes of enhancing the        electronic health record through a set of strategies and        initiatives designed and implemented across the healthcare        enterprise. These strategies and initiatives are based on        departmental and specialty workflows for all clinical imaging        content, and include methods for capture, indexing, management,        storage, access for retrieval, viewing, exchange and analytics.”

Enterprise imaging (EI) platforms typically provide the infrastructure,modalities, devices, and integration points, as well as astandards-based repository for storage of both DICOM and non-DICOMclinical images and video. Those centralized image repositories—e.g., avendor neutral archive or an enterprise wide PACS system—typicallyinclude indices of both image and metadata-information contents held inthe archive.

Medical imaging archives are increasingly becoming modality agnostic,modality vendor agnostic, specialty and service line agnostic, andviewer agnostic. Standards-based interfaces and communications,including DICOM, HL7, and standards-based Web Services, connect, enable,and support image acquisition workflows across modalities anddepartments. Image acquisition devices that support these standards maystore their images, with meta-information, into the VNA. Acquisitiondevices that are supported include departmental DICOM imagingmodalities, point-of-care acquisition modalities, handheld device photoor video apps, digital capture systems in procedure rooms, imageexchange gateways, and software designed to import content saved on adisk or received by referring or patient portals.

Clinical content and multimedia content span four broad categories ofmedical workflows within Enterprise Imaging: diagnostic imaging,procedural imaging, evidence imaging, and image-based clinical reports.

Medical workflows across many departments capture and create a varietyof types of “multimedia” information that is important to preserve,correlate with the images, and make accessible via the patient medicalrecord. Multimedia content includes waveforms, audio or video clips, aswell as other forms of graphical content that summarize imaging resultswith the results from other medical procedures and tests.Non-radiological examples can be found in many specialties includingCardiology, Neurology, Gastroenterology, Ophthalmology and Obstetrics.Graphical “report-style” results from various medical departments areincreasingly being created and saved as PDF objects. These can includeembedded images that show key findings, graphical diagrams that show thearea of interest, or other measurement or test result information thatcorrelates with the images.

Other examples of related multimedia content include time-basedwaveforms such as those produced by ECG or EEG devices. These may betreated as documents or image-like objects. Waveforms may be recordedand stored in a raw or processed form that requires an application todisplay them, or in some human-readable rendered form (like a PDF orscreenshot). Like images, waveforms too can be classified as bothevidence and diagnostic. Waveforms are the graphical representation ofdiscrete data points but may be used as the sole basis of interpretationwhen other tools for analysis of discrete data points are not availableor routinely incorporated within the interpretation protocol.

Most types of multimedia content, including waveforms, PDF reports, MPEGvideo clips, and JPEG photos, can be DICOM wrapped and stored as DICOMobjects or they can be treated as a native document type (e.g., PDF,JPEG, MPEG, etc.) and saved in systems that can manage them as nativeobjects. An important consideration is how this information will bemanaged, correlated, accessed, and viewed by physicians and patients.Wherever possible, related patient images and multimedia content couldbe made readily discoverable and shown together in a useful, naturalway.

DICOM provides support for encoding both generic identification andmodality and specialty-specific acquisition context for all enterpriseimaging modalities. DICOM-like metadata can also be added to other imagefile formats like JPEG or TIFF. Other alternatives include encapsulatingthe image in a different standard format, such as HL7 Clinical DocumentArchitecture (CDA), as is defined by the IRE Scanned Document (XDS-SD)profile, so that metadata remains directly associated with their relatedmedical images. The invention described herein supports both approachesto encapsulating and saving medical metadata together with theirassociated medical imagery.

Video Collaboration with Medical Imaging

This invention relates to a videoconferencing system for ‘live’, i.e.,real time, near real time or minimally latent, viewing of streamingmedical imagery, and more particularly, to a network system and methodsof using said videoconferencing system with both medical and non-medicalimagery, and multiple input operators (participant “cognitivecollaborants”), each viewing the other's inputs collaboratively andconcurrently.

In the past, video conferencing systems could be summarized as enablinga plurality of users systems connected to each other, each being adaptedto display a work area on a display screen or connected through acomputer network. Collaboration of work is done on each system by use ofa management table for registered node identification codes given foreach system user. That is, every computer system, or one system,requires storage of collaboration user identifier in at least one of theuser's computer system. The novelty of the current invention—a systemand methods of multimodal clinical communications, collaboration,consultation and instruction for use with medical imagery—has improvedupon prior art by allowing modular and scalable network clusters ofgateway streamer servers that enable dynamic control allowing for fasterand more efficient performance, as well as enabling multiparty cognitivecollaboration with medical imagery in a Digital Imaging andCommunications in Medicine environment, hereinafter referred to asDICOM.

The DICOM Standard pertains to the field of medical imaging informatics.The DICOM Standard is well known in the arts and facilitatesinteroperability of medical imaging equipment by specifying a set ofprotocols to be followed by devices claiming conformance to thestandard. The DICOM Standard outlines syntax and semantic of commandsand associated information which can be exchanged using these protocols.For media communication, it provides a set of media storage services tobe followed by devices claiming conformance to the DICOM Standard, aswell as a file format and medical dictionary structure to facilitateaccess to the images and related information stored on interchangemedia. DICOM data file format is data formatted in groups ofinformation, known as Data Sets. The DICOM Standard provides a means toencapsulate in a single file format structure the Data Set related to aDICOM information object. The DICOM Standard requires a single fileformat structure, as the DICOM Standard specifies that each DICOM filecontain both File Meta Information and a properly formatted Data Set (asspecified in DICOM Standard 3.10). The DICOM Standard further specifiesthat the byte stream of the DICOM Data Set be placed into the file afterthe DICOM File Meta Information (as specified in PS 3.10 DICOM Part10:Media Storage and File format for Media Interchange).

The DICOM Standard specifies the rules for encapsulating DICOM Data Setsin the requisite DICOM File format. The DICOM Standard requires that afile meta information header be present in every DICOM file, and thatthe file meta information includes identifying information of the DataSet (PS 3.7-1). The DICOM Standard requires that the Data Set conform tothe service-object pair (SOP) Class specified in the file metainformation. “The DICOM File format provides a means to encapsulate aFile the Data Set representing a SOP Instance relating to a DICOMInformation Object.” The DICOM Standard provides for the encapsulationof waveform data (PS 3.5 Part 5: Data Structures and Encoding), and forthe encapsulation of structured reports (Supplement 114: DICOMEncapsulation of Clinical Document Architecture Documents) withinimagery bit streams to facilitate the interchange of information betweendigital imaging computer systems in medical environments.

The DICOM File Meta Information includes identifying information on theencapsulated DICOM Data Set. The DICOM Standard requires that a fileheader of identifying information be present in every DICOM file. TheDICOM file header consisting of a 128 byte File preamble, followed by a4 byte DICOM prefix, followed by the File Meta Elements. This means, forexample, that a DICOM file of a chest x-ray image actually contains thepatient identification within the file, so that the image can never beseparated from patient information by mistake. A DICOM file containsboth the image and a large amount of patient information about whom,where, and how the image was acquired, known in the arts as patientmetadata.

However, DICOM files often contain little information about the contentof the imagery or meaning of the imagery pixels, the encapsulatedwaveform data used for audio clinical notes, or the encapsulatedstructured reports used for clinical documents, all of which are usedfor clinical detection, diagnosis and treatment of disease. This networksystem improves upon and applies in a collaborative environment whichprovides for capture, retrieval and concurrent viewing of both live andarchived medical imagery streams for communication, collaboration andconsultation with one or more sources of streaming imagery data by oneor more users, also known as participant cognitive collaborants.Collaborated medical imagery streams comprise one or more sources ofstreaming imagery data, including DICOM imagery files. As used herein,DICOM imagery files include modality information objects, (e.g.streaming video), waveform information objects (e.g. voice audio,echocardiogram), and structured report document information objects(e.g. clinical documents), as specified in PS 3.3 Part 3: InformationObject Definitions of the DICOM Standard.

Medical imagery streams include DICOM imagery files. This network systemallows for each user to collaborate simultaneously with all usersviewing every other users' work product, as the work product is beingcreated, all coincident with one or more streams of streaming imagerydata wherein each server manages streams of medical imagery togetherwith participant cognitive collaborant input illustrations for use withDICOM imagery files. The network system provides live video and audiocommunication, as well as a method of viewing, recording andtransmitting streaming imagery data, which include DICOM imagery files,in DICOM format, which requires a single file format structure.Streaming imagery data includes both live and archived imagery data. Asused herein, multi-channel streaming imagery data is defined as acollection of one or more sources of streaming imagery data each ofwhich comprise at least one image frame that defines a time progressionof output from various sources, which include video, encapsulatedwaveform data, and encapsulated structured reports.

The network system provides multi-channel multiplexed capability forcapture, retrieval and concurrent viewing of both live and archivedmedical imagery streams for communications, collaboration, consultationand instruction with one or more sources of streaming imagery data byparticipant cognitive collaborants. Participant cognitive collaborantinput illustrations as defined herein include, but are not limited totelestrations, drawings, sketches, text annotations, including lettercharacter text and numeric character text, image annotations, wave formannotations, voice annotations, video annotations, augmented realityimagery annotations, 3D/4D imagery annotations, outcomes annotations,costs annotations, resource consumption/utilization annotations, hapticannotations, patient metadata, imagery metadata, semantic metadata andannotations, appended patient metadata, appended imagery metadata andappended semantic metadata and annotations. The network system appendsparticipant cognitive collaborant input illustrations to streamingimagery data, and encapsulates and saves those input illustrations,together with streaming imagery data, and relevant imagery metadata andsemantic metadata and annotations, including appended imagery metadataand appended semantic metadata and annotations, from the collaborationsession in single file format structures, known as collaborated imageryfiles. The ‘single file encapsulate and save’ functionality of thenetwork system encapsulates and saves collaborated imagery files insingle file format structures, as may be required or allowed bystandards for clinical documentation or medical records storage,including those as specified in the DICOM Standard (e.g. as DICOMfiles).

The network system appends metadata tags to participant cognitivecollaborant input illustrations and encapsulates and saves those taggedinput illustrations together with the Data Set from the streamingimagery data and relevant metadata information from the metadata headerin single file format structures for use within a DICOM imageryenvironment, including those as specified in the DICOM Standard. Thenetwork system appends metadata tags to alpha-numeric text annotations,image annotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, outcomes annotations, costs annotations, resourceconsumption/utilization annotations, haptic annotations and clinicaldocuments and encapsulates those alpha-numeric text annotations, imageannotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, outcomes annotations, costs annotations, resourceconsumption/utilization annotations, haptic annotations and clinicaldocuments and saves those as DICOM files. The network system can alsoappend annotation files encapsulated as DICOM files to the Data Set forstreaming imagery data, and encapsulate them together with relevantmetadata information from the metadata header for streaming imagerydata, and save in single file format structures as collaborated imageryfiles (CIF).

Collaborated imagery files, also known as CIFs, conform to the DICOMStandard and can be stored, archived, queried, and retrieved as DICOMfiles. CIFs can be stored locally in media libraries and later retrievedfor subsequent use in collaboration sessions. CIFs conform to the DICOMStandard [3.10] and can be encrypted and/or transmitted over networksfor remote viewing, communication and collaboration. CIFs conform tospecifications of the DICOM Standard for secure encapsulation of DICOMobjects in a clinical document architecture (CDA). As such CIFs can bestored as in archives conforming to health level seven (HL7),integrating the healthcare enterprise (IHE), cross-enterprise documentsharing (XDS), cross-enterprise document sharing for imaging (XDS-I),Extensible Markup Language (XML), in Tagged Image file format (TIFF), aswell as in RDF triples and RDF/XML for metadata model specification.

CIF's can also contain encapsulated and saved haptic imagery andannotations in COLLADA-compliant dae files. COLLADA (collaborativedesign activity) is an interchange file format for interactive 3Dapplications that has been adopted by ISO as a publicly availablespecification, ISO/PAS 17506. COLLADA defines an open standard XMLschema for exchanging digital assets among various graphics softwareapplications that might otherwise store their assets in incompatiblefile formats. COLLADA documents that describe digital assets are XMLfiles, usually identified with a .dae (digital asset exchange) filenameextension.

CIFs conform to specifications of the DICOM Standard for encapsulationof audio with imagery data sets. CIFs conform to specifications to theDICOM Standard for DICOM structured reporting. CIFs can be viewed asstand-alone medical imagery, or embedded into other CIFs as video, audioand haptic annotations. The network system can create collaboratedimagery studies, also known as CIS's, which include one or morecollaborated imagery files, encapsulated and saved in single file formatstructures, as may be required or allowed by standards for clinicaldocumentation or medical records storage, including those as specifiedin the DICOM Standard format. Collaborated Imagery Studies, also knownas ‘Clini-DOCx’ are visual story boards can be used for capture,display, file exchange, publication and distribution of collections ofclinical cognitive vismemes.

The DICOM Standard defines the characteristics of a medical studyperformed on a patient as, “a collection of one or more series ofmedical images, presentation states, SR documents, overlays and/orcurves that are logically related for the purpose of diagnosing apatient. Each study is associated with exactly one patient” (PS 3.3A.1.2.2 STUDY IE). Streaming imagery data can include both collaboratedimagery files and collaborated imagery studies. Both CIFs and Clini-DOCxcan be incorporated into medical image streams of live or archivedstreaming imagery data for use during synchronous or asynchronouscollaboration sessions.

The traditional way of capturing an image from a medical imaging devicecommonly called a modality, generally consisted of an operator ortechnician first conducting a scan. Then, using the modality to save theimage, in still or motion video format, into the modality memory or intoa main image storage database. The next step in the process typicallyinvolved downloading the image into a hospital database, known in thearts as a Picture Archiving and Communications System, hereinafterreferred to as PACS or PACS server. PACS is a medical imaging technologywhich provides economical storage of, and convenient access to, imagesfrom multiple modalities (source machine types). Electronic images,including patient information known in the arts as patient metadata, aretransmitted digitally to and from PACS, eliminating the need to manuallyfile, retrieve or transport film jackets. The universal form of PACSimage file storage and transfer is the DICOM Standard, and is well knownin the arts. PACS can be further defined by a storage and managementsystem for medical images.

In the medical field, images such as x-rays, MRI's and CAT scanstypically require a greater amount of storage than other images in otherindustries. A clinician would access the PACS system to retrieve theimage, view and review the image, and conceivably develop a diagnosisbased on the information from the image. This system imagery is viewedby a user and diagnosis made without image delay and the useraccomplishes all these tasks live. “Live” referring to events simulatedby a computer at the same speed that they would normally occur in reallife. In graphics animation, for example, a live program (such as thisinventor's system) would display objects moving across the display atthe same time they would actually move, or in the case of thisinvention, a cognitive collaborant views the image live and collaboratesfrom cognitive collaborant to cognitive collaborant with no perceivabledelay to any of them.

The inventor has developed a novel and simple network system and methodsof using the same, to allow a group of cognitive collaborants toconcurrently collaborate on a computer system, with each participantcognitive collaborant viewing each other's telestrations, drawings, andannotations and saving them together with streaming imagery data,annotations and relevant imagery metadata, including appended imagerymetadata and semantic metadata and annotations, and saving them togetherin single file format structures as may be required or allowed bystandards for clinical documentation or biomedical records storage,including those as specified in DICOM, C-CDA and FHIR Standards forinteroperable health information exchange.

SUMMARY

A network system and methods for using the same for concurrentcollaboration between users, collaborating by a variety of inputillustrations, which include video, audio, telestrations, drawings andannotations, as well as collaborating on medical images that aretypically accessed on a storage server database, imaging archives, orcontinuous streaming video.

The invention relates generally to a multimedia collaborativeconferencing system and methods of using the same for generating inputillustrations, which include telestrations, drawings and annotations onmedical images concurrently with other users and saving participantcognitive collaborant input illustrations with streaming imagery data,annotations and relevant imagery metadata, including appended imagerymetadata in single file format structures, including those as specifiedin the DICOM Standard. Applicant's network system is known as the TIMSClinical Network System. It is comprised of three essential components:one called TIMS Clini-Pod Network Servers (CNS); another called TIMSClini-Ports; and a third called TIMS Clini-Docks, as depicted in FIG. 1.A Tele-Visual Imagery Informatics Management System is hereinafterreferred to as TIMS. TIMS Clini-Pod Network Servers (CNS) are computersthat manage users, security, authentication, authorization, imagestreams, channels and sessions within the TIMS Clinical Network System(i.e. the invention described herein) that allows for multiple users inmultiple locations to concurrently collaborate on the images, each userto input highlighted graphic electronic traces and annotations over themedical image, encapsulate and single file save each and all inputillustrations from participant cognitive collaborants, which includetelestrations, drawings, and annotations together with streaming imagerydata, annotations and relevant imagery metadata, including appendedimagery metadata and semantic metadata and annotations, fromcollaboration sessions in single file format structures, known ascollaborated imagery files, as may be required or allowed by standardsfor clinical documentation or medical records storage, including thoseas specified in the DICOM Standard. DICOM compliant files must containboth imagery data sets and metadata information.

TIMS Clini-Docks include a medical image acquisition system adapted forreceiving and transmitting medical images, constructed from, a computerhaving communications capability adapted for acquisition andtransmission of a plurality of medical imaging and video signals.Wherein the medical image and video signals are acquired at the medicaldevice's native resolutions, transmitting the signals at their nativeresolutions and native frame rates to a receiving device, receiving themedical imaging video signals in analog or digital form, and ifrequired, compressing and scaling the signal, converting the signal todigital form for transmission, and transmitting the digital signalsusing secure encryption protocols to a display device. TIMS Clini-Docksare capable of concurrently acquiring signals from a plurality ofmedical imaging systems, as depicted in FIG. 1, including but notlimited to, ultrasound, Computer Tomography (CT) scan, fluoroscopy,endoscopy, magnetic resonance imaging, nuclear medicine, echocardiogramultrasound and microscopy. Medical imaging equipment is also referred toas modalities. A more complete list of sources for DICOM imagery streamscan be found in the DICOM Standard [PS 3.3 Part 3: Information Objectdefinitions], which include video (imaging), audio (waveform), andclinical documents (structured reports).

TIMS Clini-Docks can also receive the video image signal from aplurality of video sources, including but not limited to, S-video,composite color and monochrome, component red blue green video (RGB,three additive primary colors), Digital Visual Interface (DVI), anyvideo transport protocol including digital and analog protocols, highdefinition multimedia interface (HDMI, compact audio video interfaceuncompressed digital data), serial digital interface (SDI), and DICOMvideo in their native, enhanced or reduced resolutions or their native,enhanced or reduced frame rates. The component, known in this inventionas TIMS Clini-Pod Network Servers (CNS), manage communications betweenall acquisition systems (TIMS Clini-Docks), between all TIMSClini-Ports, the computer workstations used by cognitive collaborantsduring collaboration sessions, between hospital servers, located on siteor remotely, that store hospital images, and hospital networks in bothlocal area and wide area configurations.

TIMS Clini-Pod Network Servers (CNS) manage both live and archivedstreaming imagery data acquired from TIMS Clini-Docks, and archivedimagery, including collaborated imagery files, retrieved inpredetermined digital single file format structures, including those asspecified in DICOM Standard, and stored locally in media libraries onparticipant cognitive collaborants computer storage devices, in imagedata repositories on tele-visual imagery informatics management systemservers, in image data repositories on cloud storage devices andlocations, in image data repositories on picture archiving andcommunications system repositories, on other image data repositoriescompliant with standards for digital imaging and communications inmedicine, or on any other data repository that allows streaming imagerydata, annotations and metadata, including semantic metadata andannotations, to be combined in native single file format structures,including in such locations as data containers and data catalogs,clinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations.

A participant or user computer can be defined as typically made ofseveral components such as a main circuit board assembly having acentral processing unit, memory storage to store programs and files,other storage devices such as hard drives, and portable memory storage,a power supply, a sound and video circuit board assembly, a display, andan input device such as a keyboard, mouse, stylus pen and the likeallowing control of the computer graphics user interface display, whereany two or more of such components may be physically integrated or maybe separate. In one depiction, a remote location communicates with thenetworked computer, for the purpose of collaborating and conferencingwith medical streaming imagery data.

TIMS Servers (CNS) manage the master control functionality of the TIMSClinical Network System. This functionality is achieved via theconnection of TIMS Servers (CNS) to TIMS Clini-Docks and allows multipleusers in multiple locations to view live all telestrations, andannotations from each of the users during collaboration sessions, asdepicted in FIG. 1. Telestrations and annotations are added as appendedlayers over the source video and do not alter the source imagery. Inaddition, when multiple TIMS Clini-Docks are connected to multiplemedical modalities, as shown in FIG. 1, TIMS Clini-Pod Network Servers(CNS) enable concurrent collaboration with each and all of thesemultiple sources of streaming imagery data. TIMS Clini-Pod NetworkServers (CNS) dynamically control which of the multiple sources ofstreaming imagery data each TIMS Clini-Port wishes to view, as depictedin FIG. 3.

TIMS Clini-Pod Network Servers (CNS) are typically deployed in severalconfigurations: Clini-Pod or pairs, typically 1-2 Clini-Pods(hub-and-spoke or peer-to-peer); Clini-Pod Quads, or teams [typically2-4 Clini-Pods); Clini-Pod Squads, (typically four Quads, or 16 Pods)and Clini-Pod Hives [4 Clini-Pod Squads]. Local CNS network serversconnect individual cognitive collaborants, also known as pod teammembers to devices in their Clini-Pod, as depicted in FIG. 13. Team CNSnetwork servers interconnect four Clini-Pods each other to allow forfour-party tele-visual communication and live synchronous collaborationwith shared work products, as depicted in FIG. 14. Hive CNS networkservers connect four or more team network servers as depicted in FIG.15. TIMS Clini-Pod Network Servers can be deployed in point-to-point,hub-and-spoke and mesh chord networks, as depicted in FIG. 16, as wellas in other network configurations described in the BellcoreTelecommunications Management Network [TMN] architecture. In particular,TIMS Clini-Pod Network Servers can be deployed in core-spine-leafnetwork topologies, as depicted in FIG. 17, as well as in 2-tier, 3-tieror N-tier application architectures, as depicted in FIG. 18.

TIMS Clini-Port software applications allow participant cognitivecollaborants to add other sources of streaming imagery data by selectingthe “add+” channel selection tab, and viewed on the channel tabs of themulti stream viewer as shown in FIG. 3, (channel 1X . . . ). Themulti-channel stream view capability of TIMS Clini-Port softwareapplications allow concurrent viewing of multiple channels of both liveand archived medical imagery streams as depicted in FIG. 7. Themulti-channel stream view selection capability is depicted in FIG. 9,and again in FIG. 10 with multiple channels of both live (“stream”) andarchived (image “81420095249.jpg, and image “99200982617.mpg”) medicalimagery streams selected for capture, retrieval and concurrent viewingduring a collaboration session. TIMS Clini-Port software applicationsinclude DICOM view capability, which allows participant cognitivecollaborants to view, communicate, collaborate, consult and instructwith DICOM imagery streams. TIMS Clini-Port software applicationsinclude capabilities to view non-DICOM imagery as well, which allowsparticipant cognitive collaborants to view, communicate, collaborate,consult and instruct with non-DICOM imagery streams. The multi-channelstream view capability of TIMS Clini-Port software applications allowsparticipant cognitive collaborants to capture, retrieve and concurrentlyview both live and archived medical imagery streams for communication,collaboration, consultation and instruction with one or more sources ofstreaming imagery data by one or more participant cognitivecollaborants, with both DICOM and non-DICOM imagery streams duringcollaboration sessions. Each participant cognitive collaborant, some ofwhom may be located remotely to the imaging modalities, is able to view,analyze, discuss and comment on each of the input illustrations fromparticipant cognitive collaborants concurrently, live, and save suchanalysis or discussion as may be clinically relevant.

In one embodiment, the connection of TIMS Clini-Pod Network Servers(CNS) to TIMS Clini-Docks allows TIMS Clini-Ports to customizepreferences for capture, retrieval, and viewing of streaming imagerydata while the patient is still on the examination table. TIMSClini-Ports can have direct access and control of the streaming imagerydata and maintain the native resolution and frame rate output from themedical modality. If desired, TIMS Clini-Ports can adjust the nativeresolution, frame rate, and compression of the streaming imagery dataspecific to the user's preferences. In addition, TIMS Clini-Ports areable to live instruct clinicians who are controlling streaming imagerydata at their respective modality sources, and view the results of thoseinstructions to ensure that imagery acquired is consistent with userpreferences, as depicted in FIG. 3. Those instructions are conveyed viatwo way communication between user and clinician with voice, text, videoor telestrations within the TIMS Clini-Pod Network System and are notreliant upon any external communications network.

Without access to master control of TIMS Clini-Docks by TIMS Clini-PodNetwork Servers (CNS), imagery viewed by a remote client using anotherinvention is limited to the quality of the view and capture settingsspecified by others, which may be different than those desired orrequired by the remote client. TIMS Clini-Docks are multichannelstreamer stacks that allow live capture and archived retrieval fortele-visual communications with: (1) streaming video; (2) medicalimagery modalities and waveforms; (3) electronic medical records; and(4) clinical and multiomic maps and biometric data streamvisualizations.

As used herein, “streaming medical imagery” includes all informationobjects described in the DICOM Standard, including images, video,modality imaging and waveforms—audio, visual and haptic wave forms andfiles, medical records and clinical documents, multiomic—phenotypic,genomic, metabolomic, pathomic, radiomic, radiopathomic andradiogenomic—maps and clinical data sets, including those as describedin the Institute of Medicine's Towards Precision Medicine—A New Taxonomyof Disease; along with biometric data stream visualizations fromconnected medical devices, signals and sensors used for local and remotepatient monitoring.

In one embodiment TIMS Clini-Docks can be deployed in four (4) dualchannel streamer stacks to accommodate both live and archived streamingimagery data from these four principal modalities for tele-visualcommunications and collaboration with imagery informatics. Clini-Dockstreamer Channel (1) is typically reserved for video communications andconferencing among team members and other cognitive collaborants.Channel (2) normally designated for electronic medical records andpatient monitoring; Channel (3) for medical imaging modalities and waveforms. Channel (4) for data mapping and interactive biometric datastream visualizations, including virtual reality and augmented realitydisplays. The TIMS Clini-Port typically has one or more multi-channelmonitors for connected devices, which can be situated locally, withinthe Clini-Pod, or at remote locations, including other Clini-Pods.

TIMS Clini-Docks, due to its novel capabilities, can acquire analog ordigital video signals, standard or non-standard video resolutions,medical or non-medical imagery, live or archived imagery, and compressedor uncompressed imagery formats. TIMS Clini-Docks converts analogsources of streaming imagery data, as well as non-standard sources ofstreaming imagery data into digital imagery data sets for use byparticipant cognitive collaborants during collaboration sessions. TIMSClini-Docks can also convert non DICOM digital imagery data sets,including non DICOM modality imaging (e.g. video), waveform data (e.g.voice, audio, haptic), and structured reports (DICOM-SR from PACS) andclinical documents (CCD, CCR from EHR medical records systems) intoDICOM imagery streams for use by participant cognitive collaborantsduring collaboration sessions. The TIMS Clini-Dock stack depicted inFIG. 1 allows for capture of multiple sources of streaming imagery datain any and all combinations of the preceding specifications, (e.g. bothDICOM and non-DICOM imagery streams, standard and non-standard imagerystreams, and compressed and uncompressed imagery streams) and allowsTIMS Clini-Ports concurrent viewing of multiple sources of streamingimagery data. TIMS Clini-Docks incorporate approved medical devicecomponents that processes any video output from a video source into animage stream, including but not limited to streaming imagery data frommedical modalities, as depicted in FIG. 1.

TIMS Clini-Docks are connected directly to multiple sources of streamingimagery data, as depicted in FIG. 1, and continuously streams images toTIMS Clini-Pod Network Servers (CNS). Any number of TIMS Clini-Ports canrequest information from a TIMS Clini-Pod Network Servers (CNS). EachTIMS Clini-Port in a conference with another or other TIMS Clini-Portscan view all the TIMS Clini-Port object inputs as they occur. TIMSClini-Ports refer to computer workstations used by cognitivecollaborants during collaboration sessions, typically for medical reviewand diagnosis of patient image data.

TIMS Clini-Pod Network Servers (CNS) keep track of all TIMS Clini-Docksthat have image streams available and displays a list of image streamsavailable TIMS Clini-Ports, as depicted in FIG. 3. TIMS Clini-PodNetwork Servers (CNS) communicate with image repositories, including butnot limited to PACS system repositories, and store information on allTIMS Clini-Ports' computers live. TIMS Clini-Pod Network Servers (CNS)include software components that manage streaming requests to TIMSClini-Docks; manage authentication and authorization tasks for accessand privileges; manages users information, roles, session logs and,configurations for TIMS Clini-Pod Network Servers (CNS) and TIMSClini-Docks; manage web services interactions with TIMS Clini-Ports;send, query and retrieve collections of one or more streaming imagerydata files, including collaborated imagery files, also known as studies,to and from image repositories, as depicted in FIGS. 10 and 11,including but not limited to DICOM compliant image repositories, e.g.,PACS; specify unique combinations of image quality, resolution,compression and frame rates as may be required for each collaborationsession, as depicted in FIG. 3; access patient information from a DICOMModality Worklist utility (DMWL); collaborated imagery files, to TIMSClini-Ports; manage text chat information; manage DICOM send services,wherein the DICOM send service sends the annotated images to PACS orDICOM compliant image repositories, also known as medical imagearchives, as depicted in FIG. 10; allow for query and retrievefunctionality that retrieves lists of DICOM studies from PACS serversand DICOM compliant image repositories and sends those studies to TIMSClini-Ports.

A DICOM study is defined as a collection of one or more medical imagesand patient data combined in single file format structures, includingthose as specified in the DICOM Standard. DICOM Modality Worklist isdefined as a software utility that invokes DICOM query and retrievefunctionality which enables imaging equipment (e.g. medical modalities)to query medical image stores, including but not limited to PACS, andobtains details of patient and scheduled examinations electronically,including patient demographics and study data, avoiding the need to typepatient information multiple times, as depicted in FIG. 10. Clini-Podstypically deploy with Clini-CDR (Clinical Data Repositories, consistingof p-CKR [personalized Clinical Knowledge Repositories] with localstorage of CIFs and clinical cognitive vismemes in personalized clinicalknowledge repositories, clinical cognitive vismeme vaults; and metadatarepositories which house reference links to collaborated imagery files,along with dicomized cognitive collaboration security tokens whichprovide granular control over access to shared imagery files stored inclinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories.

Security tokens for cognitive collaboration, selectively providingaccess to imagery information objects, their associated metatdata andannotations, including personal health information (PHI), can becreated, read, updated or deleted, with concurrence by or among one ormore participant cognitive collaborants, before, during or aftercognitive collaboration s sessions. Security tokens for cognitivecollaboration can be maintained in separate metadata repositories,including blockchain metadata repositories and blockchain data ledgers,and accessed by cognitive collaborants with appropriate securityprivileges.

TIMS Clini-Pod Network Servers (CNS) also manage all the participantcognitive collaborant input illustrations, specifically, the entireparticipant cognitive collaborant input illustrations, sketches,drawings, telestrations and annotations. Participant cognitivecollaborant input illustrations as previously defined herein include,but are not limited to telestrations, drawings, sketches, textannotations, including letter character text and numeric character text,image annotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, outcomes annotations, costs annotations, resourceconsumption/utilization annotations, haptic annotations, imagerymetadata and appended imagery metadata, as depicted in FIG. 7. Allparticipant cognitive collaborant input illustrations are managed byTIMS Clini-Pod Network Servers (CNS) based on a file sharing schemewhere new input illustrations keep getting appended to files on TIMSClini-Pod Network Servers (CNS). TIMS Clini-Pod Network Servers (CNS)distribute copies of streaming imagery data to each of the participantcognitive collaborants. Since participant cognitive collaborantscollaborate only with copies of images, they do not alter the originalstreaming imagery data in any way. This approach of generating inputillustrations on TIMS Clini-Pod Network Servers (CNS), and distributingonly those input illustrations and not the underlying images to eachparticipant cognitive collaborant, significantly improves operatingperformance and reduces image latency and wait times. That method ofmoving images with illustrations back and forth from a computer to aserver, results in losing illustration quality or consuming morebandwidth. However, with this novel invention, the process ofmulti-layer multi user input illustrations on any underlying images,including streaming imagery data, and updating and appending on thestreaming imagery data with multiparty annotations and metadata withoutsacrificing network bandwidth, is novel to this invention.

TIMS Clini-Pod Network Servers (CNS) allow TIMS Clini-Ports to createcollaborated imagery files synchronously or asynchronously. TIMSClini-Pod Network Servers (CNS) use a process of local registration toidentify the image frames needed for viewing on each of the participantcognitive collaborant computers, and sends to each of them only theimage frames necessary for participation in a collaboration session.TIMS Clini-Pod Network Servers (CNS) enable each participant cognitivecollaborant to use a scalable window so all input illustrations for eachand every participant cognitive collaborant are dynamically ratio metricbased on the underlying image aspect ratio of the respective participantcognitive collaborant computer. Therefore, all the input illustrationsalways point to the part of the window and image as originally intended,regardless of window size on the clients computer display. A centralframe counter originating in the participant cognitive collaborantcomputer, which has play/pause control, issues frame synchronizationcommands to synchronize the image streams on all participant cognitivecollaborants' computers. This method significantly reduces bandwidthrequirements and improves responsiveness of system updates and imageryappends. Each client computer which has play/pause control also sendssynchronizing commands whenever its displayed images are paused. Thisensures that the same frame is available to all participating clients bybroadcasting that pause frame number along with the pause command to allparticipating clients.

Client participants can receive video streams directly from TIMSClini-Docks using a local area network. The invention can also detect ifa user has low bandwidth, in transmission, or in reception, or in bothand can compensate by only sending selected image frames to that user.For example, with low bandwidth TIMS Clini-Pod Network Servers (CNS) cansend every third, fifth, or Nth frame of a collaborated imagery toclients so that client does not have any perceptible delay. Remoteclient participants using the internet must receive all imagery fromTIMS Clini-Pod Network Servers (CNS) for secure transmission, ratherthan directly from local TIMS Clini-Docks, to ensure streaming imagerydata is not transmitted over the internet without encryption.

TIMS Clini-Ports, also known as participant cognitive collaborants, cantake several roles. Each participant cognitive collaborant can capture,retrieve and concurrently view both live and archived streaming imagerydata of their own choosing, including medical imagery streams selectedfor the collaboration session; capture, retrieve and concurrently viewboth live and archived streaming imagery data streams selected by otherparticipant cognitive collaborants, including medical imagery selectedfor the collaboration session; each participant cognitive collaborantcan add multiple sources of streaming imagery data, also referred to asmultiple channels, of both live and archived streaming imagery data forother participant cognitive collaborants to capture, retrieve andconcurrently view; capture, retrieve and concurrently view multiplesources of both live and archived streaming imagery data, includingmedical imagery streams selected for a collaboration session;concurrently add input illustrations on both live and archived streamingimagery data; taking on any and all of the above roles dynamically, asdepicted in FIG. 4.

In addition, TIMS Clini-Port software applications are collaborative,interactive; tools for synchronous or asynchronous media annotation,which can be used with medical files to enable participant cognitivecollaborants to communicate, collaborate, consult and instruct withmedical images for clinical review and discussions and deciding onrelevant medical procedures.

This novel invention—combination streamer-splitter-server-router-networkgateway servers—allows any of the TIMS Clini-Ports to host acollaboration session with any other TIMS Clini-Port, in various networkconfigurations, including peer-to-peer, hub-and-spoke, mesh chordnetworks, as depicted in FIG. 16. A collaboration session host selectsany number of participant cognitive collaborants from their contactlist, as depicted in FIG. 5, and sends a request to those clients withwhom they wish to collaborate. Each participant cognitive collaborantreceiving the request can elect to join or decline the session byselecting the appropriate button on the dialog box that appears on theircomputer monitor, as depicted in FIG. 6. Upon acceptance of the request,the cognitive collaborant client's monitor is automatically switched toview the same imagery as the collaboration session host. The host canselect live streaming imagery data from any of the available TIMSClini-Docks, as depicted in FIG. 3, can also select from any archivedstreaming imagery data available through the query and retrievefunctions, as depicted in FIG. 11, and concurrently collaborate usingall selected imagery data streams—live, archived or both—for multimodalclinical communications, collaboration, consultation or instruction withall participant cognitive collaborant clients during the collaborationsession.

All input illustrations added by participant cognitive collaborants areconcurrently visible to all of the other participant cognitivecollaborants. In addition, each participant cognitive collaborant canadd input illustrations, which include telestrations, drawings, textannotations, image annotations, wave form annotations, voiceannotations, video annotations, augmented reality imagery annotations,3D/4D imagery annotations, outcomes annotations, costs annotations,resource consumption/utilization annotations, haptic annotations, tostreaming imagery data, together with relevant imagery metadata,including appended imagery metadata and semantic metadata andannotations. Furthermore, each participant cognitive collaborant clientcan also use the TIMS Clini-Pod Network System to chat with each otherduring a collaboration session using a text chat facility. A separatetext window box is displayed that allows for each participant cognitivecollaborant to instant message each other in text format and includethose images as input illustrations, as depicted in FIG. 7. One featureof this invention is that the host can disable the edit control of anyclient, such that a particular client will not be able to add or editthe annotations or telestrations, as depicted in FIG. 8. At this point,the client can only view the annotations made by others. The host canalso pass the control of the video stream start/stop/pause functions toanother client. This control allows the host to enable or disable thefunctionality to all clients or selected clients and can be done at anytime during the collaboration session. At the conclusion of the session,participant cognitive collaborants can encapsulate and save all inputillustrations, which include telestrations, drawings and annotationstogether with streaming imagery data, and relevant imagery metadata,including appended imagery metadata and semantic metadata andannotations, from the collaboration session, in single file formatstructures, known as collaborated imagery files. Collaborated ImageryFiles are encapsulated and saved in single file format structures, asmay be required or allowed by standards for clinical documentation ormedical records storage, including those as specified in the DICOMStandard (e.g., as DICOM files). Participant cognitive collaborantclients can send collaborated imagery files to any PACS or DICOMcompliant image data repository, or send to any other data repositorythat allows streaming imagery data, annotations and metadata, includingsemantic metadata and annotations, to be combined in native single fileformat structures, including in such locations as data containers anddata catalogs, clinical data repositories, personalized clinicalknowledge repositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations. A session log is recorded and saved on the TIMS server, asdepicted in FIG. 9.

The invention also works with wearable signals, sensors, devices andmonitors, collectively “mobile computing”, also known as PersonalDigital Assistants. Participants (PDA) clients can use these PDAs toview, consult and collaborate on DICOM images. Personal digitalassistant is any small mobile hand held device that provides computingand information storage such as hand held computers, phones, mediadisplay devices and handheld computers, including watches and vison.

The invention enables both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instructionbetween and among participant cognitive collaborants, includingheterogeneous networked teams of persons, machines, devices, neuralnetworks, robots and algorithms. And specifically, the invention enablesmultimodal clinical communications, collaboration, consultation andinstruction during various stages of medical disease and injurymanagement, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs.

In one embodiment the invention provides for a uniquestreamer-splitter-server-router functional combination in a singlenetwork edge device, a neurosynaptic network node, having bi-directionalcommunications capability with other Pod network gateway servers vianetwork, video and wireless connectivity. Network gatewayservers—neurosynaptic network nodes—can be combined in variousmultichannel multiplexed combinations, including pod pairs (2), podquads (4), pod squads (16), and pod hive clusters (64), as depicted inFIGS. 13, 14 and 15. Neurosynaptic network servers can be deployed invarious network architectures, including point-to-point (peer-to-peer),hub-and-spoke, and mesh chords, as depicted in FIG. 16. Neurosynapticnetwork servers can be deployed can be deployed in core-spine-leafnetwork topologies, as depicted in FIG. 17, as well as in 2-tier, 3-tieror N-tier application architectures, as depicted in FIG. 18. Theseembodiments allow for dynamic neurosynaptic connectivity formultichannel multiplexed networked visual communications.

In another embodiment the invention provides a method for recursivecognitive enrichment and collaborative knowledge exchange between andamong cognitive collaborants, including heterogeneous networked teams ofpersons, machines, devices, neural networks, robots and algorithms.Specifically it provides neurosynaptic network connectivity enablingboth synchronous and asynchronous multimodal clinical communications,collaboration, consultation, instruction, that includes viewing,curating, annotating and tagging, using one or more sources ofmultichannel, multiplexed heterogeneous streaming imagery data,including both medical and non-medical streaming imagery data.

This embodiment also provides a method for rapid, adaptive deep learningand specialist skills acquisition by and among cognitive collaborants,including heterogeneous networked teams of persons, machines, devices,neural networks robots and algorithms, with neurosynaptic networkconnectivity enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation, instruction, thatincludes viewing, curating, annotating and tagging, using one or moresources of multichannel, multiplexed heterogeneous streaming imagerydata, including both medical and non-medical streaming imagery data,together with images, video, modality imagery, waveforms, audio andhaptic files, multiomic—phenotypic, genomic, metabolomic, pathomic,radiomic, radiopathomic and radiogenomic—maps and clinical data sets,biometric maps and movies, hapmaps, heat maps, data streamvisualizations, structured reports, interactive media reports, clinicaldocuments and key performance indicators during various stages ofmedical disease and injury management, including detection, diagnosis,prognosis, treatment, measurement, monitoring and reporting, as well asworkflow optimization with operational analytics for outcomes,performance, results, resource utilization, resource consumption andcosts.

A medical imagery stream is defined as a collection of one or moresources of streaming imagery data which comprise at least one imageframe that defines a time progression of output from a video source.TIMS Clini-Docks maintain image quality from source modalities asrequired for conformance to DICOM Standards for clinical use. TIMSClini-Docks specify streamer components that have secured regulatoryclearances for transmission and viewing of medical imagery streams forclinical diagnostic purposes.

In one embodiment, TIMS Clini-Pod Network Servers (CNS) provide livevideo and audio communications, as well as a method of recording,transmitting and saving images in single file format structures,including those as specified in the DICOM Standard. DICOM is a medicalimaging standard common in the medical industry. DICOM can also bedefined as a standard in the field of medical informatics for exchangingdigital information between medical imaging equipment (such asradiological imaging) and ensuring interoperability with other systems.DICOM, including protocols for device communication over a network,syntax and semantics for commands and associated information that can beexchanged using protocols, a set of storage services and devicesclaiming conformation to the standard, as well as file format andmedical directory structures to facilitate access to images and relatedinformation stored on media that shares information. The embodiment canserve as the connection point between any medical imaging modality and ahospital PACS, medical archive or other image repository, includingclinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories.

One component of this invention, TIMS Clini-Pod Network Servers (CNS),are able to connect DICOM equipment and older non-DICOM equipment to ahospital network, allowing imaging studies to be stored and saved. TheTIMS Clini-Pod Network System, this invention described herein, brieflydescribed as a trace overlay and annotation system that users cancollaborate with each other live, each viewing each other's objectinputs and those object inputs can be encapsulated and saved in singlefile format structures, including those as specified in the DICOMStandard, in PACS, in a DICOM compliant image archives, or in otherimage repositories.

Another embodiment the TIMS Clini-Pod CNS network system can be deployedas collaboration portals for multi-party cognitive collaboration amongspecialist providers; care coordination for caregiving teams both localand remote; and patient provider engagement, the support of meaningfuluse goals and objectives for electronic medical records. Clini-Pod CNSalso support health information exchange for integrated deliverysystems; for biomedical, clinical and genomic mapping and multisensorydata stream visualizations, as well as clinical decision support forvalue care-giving teams.

Still other embodiments provide networked informatics connectivity formedical kiosks, offices and retail clinics, ambulatory care and nursingfacilities. Often these facilities have limited connectivity for accessto hospital-based electronic medical systems. In those circumstances theTIMS Clini-Pod CNS as “LAND” [Local Adapter for Network Distribution]and “SEE” [Surrogate Electronic Health Record Environment] to facilitatehealth information exchange with hospitals and other caregivingfacilities. Use of TIMS Clini-Pod access and enable groups with accessto EHR systems to share electronic medical information with those who donot, and specifically by health information exchange with ConsolidatedClinical Document Architecture (C-CDA”) compliant documents, includingContinuity of Care Documents (CCD, CCD+, etc.), Fast HealthcareInteroperability Resources (“FHIR”) and Universal Transfer Forms (UTF.)

BRIEF DESCRIPTION OF DRAWINGS

Other objects, features, and advantages will occur to those skilled inthe art from the following description of an embodiment and theaccompanying drawings, in which:

FIG. 1, depicts a block diagram of the invention.

FIG. 2, depicts a block diagram of a portion of the system.

FIG. 3, depicts a graphic user interface screen shot of a cognitivecollaborant workstation: client imagery source selection display.

FIG. 4, depicts a graphic user interface screen shot of cognitivecollaborant workstation: client source imagery with illustration toolbar and collaboration function.

FIG. 5, depicts a graphic user interface screen shot of a cognitivecollaborant workstation: client selecting participant cognitivecollaborants for collaboration session.

FIG. 6, depicts a graphic user interface screen shot of cognitivecollaboration session initiation.

FIG. 7, depicts a graphic user interface screen shot of a cognitivecollaboration session, including streaming medical imagery withannotations.

FIG. 8, depicts a graphic user interface screen shot of a cognitivecollaborant workstation: client assignment of control to participantcognitive collaborant.

FIG. 9, depicts a graphic user interface screen shot list of multiplecognitive collaboration sessions.

FIG. 10, depicts a graphic user interface screen shot of patient imagestudy information.

FIG. 11, depicts a graphic user interface screen shot of patientelectronic medical record information.

FIG. 12, depicts a graphic user interface screen shot of administrativecontrols.

FIG. 13, depicts deployment of a hub-and-spoke device cluster for eitherserver-based or peer-to-peer networks.

FIG. 14, depicts a 4-Party Team Network Server cluster interconnectingwith four Clini-Pod hub-and-spoke device clusters.

FIG. 15, depicts a 4-Party Hive Network Server cluster interconnectingwith four Team Network Server clusters.

FIG. 16, depicts Alternative Network Architectures for Clini-Poddeployment: point-to-point vs hub-and-spoke vs chord.

FIG. 17, depicts traditional 3-Tier Application Architectures versusCore-Spine-Leaf Network Topology.

FIG. 18, depicts traditional 1-Tier, 2-Tier, 3-Tier and N-TierApplication Architectures.

FIG. 19, depicts Value Chain Knowledge Exchange.

FIG. 20, depicts processes for generating insights from KnowledgeMapping and Interactive Data Visualization (data analytics, aggregationand contextualization).

FIG. 21, depicts transforming Data into Information, then intoKnowledge, Wisdom, Decision and Action.

FIG. 22, depicts Information Optimization through Descriptive,Diagnostic, Predictive and Prescriptive Analytics.

FIG. 23, depicts Cognitive Value Creation with Information Optimizationand Advanced Data Analytics.

FIG. 24, depicts Adaptive Systems Learning with Augmented Analytics.

FIG. 25, depicts increasing Business Intelligence and ActionableInformation with semantic metadata and annotation.

FIG. 26, depicts domain-specific semantic search, ontology mapping andvisualization with RDF metadata.

FIG. 27, depicts semantic interoperability with metadata registries andinformation model annotation.

FIG. 28, depicts a semantic data lake for clinical, financial andoutcomes data integration.

FIG. 29, depicts building semantic data trails with metadata extractionfrom structured, semi-structured and unstructured data, includingbiomedical data from medical imaging modalities.

FIG. 30, depicts semantic metadata linking open and proprietarypharmaceutical data sets for clinical trials management.

FIG. 31, depicts semantic metadata connecting an information ecosystemwith open and proprietary clinical data sets for pharmaceuticaldevelopment and pipeline management.

FIG. 32, depicts Visualizing Value Care: Connecting Doctors, Devices,Documents and Streams—Visualizing Value Care for CollaborativeCare-Giving Teams.

FIG. 33, depicts a New Paradigm for Collaborative Value Care withCognitively-enriched Enterprise Imaging.

FIG. 34, depicts Cognitively-enriched Enterprise Imaging: Diagnostic,Procedural and Evidence Imaging, along with Imaged-Based ClinicalReports.

FIG. 35, depicts Cognitively-enriched Enterprise Imaging RepositoryInformation Architecture.

FIG. 36, depicts Cognitively-enriched Enterprise Imaging—Best PracticesWorkflow.

FIG. 37, depicts Streaming Analytics Architecture for Hospital-basedEnterprise Imaging.

FIG. 38, depicts Imagery Document Exchange with Metadata Registries andan Enterprise Imaging Data Repository.

FIG. 39, depicts Biomedical Knowledge Exchange with AugmentedIntelligence Networks.

FIG. 40, depicts Integrative Systems Biology with Multimodal,Multi-Scalar Visual Bioinformatics.

FIG. 41, depicts Augmented Pattern Recognition with MultimodalRadiogenomic Imagery and Adaptive Mind-Machine Learning.

FIG. 42, depicts Pattern Matching Algorithms for Multiple Classes ofOncology Images.

FIG. 43, depicts Early Disease Detection with Multimodal, Multi-ScalarBiomedical Sensors.

FIG. 44, depicts Clinical Knowledge Networks Integrating BiomedicalResearch with Clinical Medicine, “from Bench to Bedside”.

FIG. 45, depicts Value Drivers for Biomedical Knowledge Exchange withAugmented Intelligence Networks.

FIG. 46, depicts Molecular Profiling with Predictive Prognostic Markersfor Precision Cancer Medicine.

FIG. 47, depicts Cancer LINQ—A Learning Intelligence Network ConnectingPatients, Providers and Researchers with Biomedical Data and Knowledge.

FIG. 48, depicts Connecting Collaborative Partnerships for MultiomicData Analysis and Personalized Precision Medicine.

FIG. 49, depicts Precision Diagnostics and Precision TargetedTherapeutics Information Sciences for Personalized Precision Medicine.

FIG. 50, depicts Clinically Actionable Information from Big Data as thefoundation for Personalized Precision Medicine.

FIG. 51, depicts “See One. Do One. Teach One.” Surgical Telementoring,Teamwork and Training.

FIG. 52, depicts Imagery Guided Computer Assisted Surgery.

FIG. 53, depicts Informatics-Enriched Robotic Assisted Surgery.

FIG. 54, depicts Streaming Augmented Reality Surgical Instruction.

FIG. 55, depicts Surgical Navigation and Guidance with 3D DataVisualization and Streaming Augmented Reality.

FIG. 56, depicts Visualizing the Surgical Site for Robotic AssistedIntervention.

FIG. 57, depicts an Imagery Guided Minimally Invasive Surgical RoboticSystem.

FIG. 58, depicts Visio-Spatial Algorithms Development for PrecisionGuided Surgery.

FIG. 59, depicts Live Surgical Demonstration with Expert Panels asCollaborative Teaching Tools.

FIG. 60, depicts Live Remote Intraoperative Telesurgical Consultationduring Aneurysm Repair.

FIG. 61, depicts Live Remote Surgical Telementoring, Teamwork & Trainingwith Interactive Streaming Video and Multisensory Augmented Reality.

FIG. 62, depicts interconnected Ecosystems of the Future forInformatics-Enriched Imagery Guided Interventions.

FIG. 63, depicts various techniques for Machine Learning with MedicalImaging.

FIG. 64, depicts a Framework for Cancer Metastasis Detection with DeepLearning Models and Whole Slide Imaging.

FIG. 65, depicts visualization of Tumor Region Detection withSlide/Heatmap Overlays.

FIG. 66, depicts Pancreatic Cancer Computer Assisted Detection withConvolutional Neural Networks.

FIG. 67, depicts Pulmonary Embolism Identification with MachineLearning.

FIG. 68, depicts Bone Age Assessment with Deep Learning Systems.

FIG. 69, depicts Video-based Attributes Labeling and SemanticIdentification.

FIG. 70, depicts Data Extraction for Training Machine Learning Systemswith Medical Outcomes.

FIG. 71, depicts illuminating “black-box” understanding of MachineLearning results developed from Neural Networks [e.g., XAI—ExplainableArtificial Intelligence].

FIG. 72, depicts Convolutional and Recurrent Neural Networks with LongShort Term Memory [LSTM] in Medical Imaging.

FIG. 73, depicts Data Mining, Training and Labeling with AnnotatedMedical Images using Convolutional and Recurrent Neural Networks withLSTM.

FIG. 74, depicts a Periodic Table of Artificial Intelligence with“Elementary” PAIR Techniques [Perceive-Assess-Infer-Respond].

FIG. 75, depicts implementing Data-Information-Knowledge Networks withMachine Learning for BioIntelligence.

FIG. 76, depicts various biomedical applications for Nanorobotics.

FIG. 77, depicts several typical features of Nanorobots.

FIG. 78, depicts monitoring Nanorobotic agents designed to treat cancer.

FIG. 79, depicts medical micro robots actuated by clinical Millscanners.

FIG. 80, depicts Personalized Precision Targeted TheranosticNanomedicine.

FIG. 81, depicts Imagery Guided Precision Theranostics with TargetedDrug Payloads.

FIG. 82, depicts Nanoparticle-based Imaging Diagnostics and Therapy.

FIG. 83, depicts Multifunctional Nanoparticles for TheranosticNanomedicine.

FIG. 84, depicts Integrating Theranostics Techniques with MolecularImaging Modalities.

FIG. 85, depicts Drug Delivery, Cell Destruction and Micro-Surgery within vivo Imaging Theranostics.

DETAILED DESCRIPTION

A network system 1 for allowing users to concurrently communicate live;concurrently collaborate live, concurrently consult live, andconcurrently instruct live while concurrently viewing multiple sourcesof streaming imagery data 13 on a display screen using sketched andannotated participant cognitive collaborant input illustrations overstreaming imagery data 13 among a group of remotely located participantcognitive collaborants 10, including heterogeneous networked teams ofpersons, machines, devices, neural networks, robots and algorithms.

The network system having at least one or more TIMS Clini-Pod NetworkServers (CNS) 2 including associated data bases in communication withlocal area networks 3, in some circumstances connected to and havingaccess to a medical PACS server 4 including associated database allcapable of using the protocols required by the DICOM Standard and allhaving access to DICOM modality work list utilities for appendingimagery metadata 5 including associated databases providing medicalpatient metadata, as well as imagery metatdata, semantic metadata andannotations, and archived annotated imagery. To collect streamingimagery data 13 the system together with at least one TIMS Clini-Dock 6in contact with the local area network 3 wherein the TIMS Clini-Dock 6is providing live streaming imagery data to the local area network 3 asit receives concurrent sources of live streaming imagery data 6 frommultiple medical modalities 7, 8, 9 such as, but not limited to,ultrasound, fluoroscopy and video. A participant cognitive collaborantcan view streaming imagery data 13 in single file format structures,including those as specified in the DICOM Standard together withparticipant cognitive collaborant input illustrations 18 which include,telestrations 21, drawings 22 and annotations 234 (known herein as inputillustrations from participant cognitive collaborants) over thestreaming imagery data and saving that streaming imagery data, relevantimagery metadata, including appended imagery metadata and semanticmetadata and annotations, together with input illustrations fromparticipant cognitive collaborants 18 in single file format structures,including those as specified in the DICOM Standard, locally in medialibraries or image data repositories on their respective computerstorage devices, in image data repositories on TIMS Clini-Pod NetworkServers (CNS) 2, in image data repositories on cloud storage devices andlocations, in image data repositories on picture archiving andcommunications systems PACS 4 or in other image data repositoriescompliant with standards for digital imaging and communications inmedicine, or in any other data repository that allows streaming imagerydata, annotations and metadata, including semantic metadata andannotations, to be combined in native single file format structures,including in such locations as data containers and data catalogs,clinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations.

This invention allows for one or more TIMS Clini-Ports 10 toconcurrently use the network system at the same time. The network system1 also allows participant cognitive collaborants to concurrentlycollaborate live, as defined by this system. The plurality of TIMSClini-Ports can concurrently view multiple sources of live and archivedstreaming imagery data-13, and concurrently create input illustrations18 over that streaming imagery data 13 which include telestrations 21,drawings 22 and annotations 23, as they are appended to that imagery,and encapsulate and save those participant cognitive collaborant inputillustrations, including telestrations, drawings, and annotations,together with streaming imagery data, and relevant imagery metadata,including appended imagery metadata, from the collaboration session insingle file format structures, known as collaborated imagery files. Thenetwork system 1 ‘single file encapsulate and save’ functionalityencapsulates and saves collaborated imagery files in single file formatstructures, as may be required or allowed by standards for clinicaldocumentation or medical records storage, including those as specifiedin the DICOM Standard, Clini-Pod Network locally in media libraries orimage data repositories on their respective computer storage devices, inimage data repositories on TIMS Clini-Pod Network Servers (CNS) 2, inimage data repositories on cloud storage devices and locations, in imagedata repositories on picture archiving and communications systems PACS 4or in other image data repositories compliant with standards for digitalimaging and communications in medicine, or in any other data repositorythat allows streaming imagery data, annotations and metadata, includingsemantic metadata and annotations, to be combined in native single fileformat structures, including in such locations as data containers anddata catalogs, clinical data repositories, personalized clinicalknowledge repositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations.

TIMS Clini-Ports can retrieve archived collaborated imagery files foruse during current or future collaboration sessions. TIMS Clini-Portscan include collaborated imagery files in patient studies. In oneembodiment, a collaboration session can include one or more participantcognitive collaborants that can utilize personal digital assistants(PDA) over the internet 12.

A method for allowing one or more participant cognitive collaborants toconcurrently collaborate live on medical images 13, all participantsclients running substantially the same TIMS Clini-Port softwareapplication programs on each of the participant cognitive collaborant'scomputers; storing the programs on each of the participant cognitivecollaborant's computers. Each participant cognitive collaborant computerdisplaying the graphic user interface output 25 of those programs ontheir computer display. Each participant cognitive collaborant computerlinking to each other and to TIMS Clini-Pod Network Servers (CNS) 2using local area networks 3. All TIMS Clini-Ports 10 have access tolocal area networks 3 and internet 12. TIMS Clini-Pod Network Servers(CNS) 2 providing authentication and authorization to each participantcognitive collaborant wherein linking the participant cognitivecollaborant to DICOM Modality Worklist utilities 5, to image datarepositories connected to picture archiving and communications systemsvia PACS servers 4, to other image data repositories compliant withstandards for digital imaging and communications in medicine DICOM, toimage data repositories connected via internet 12 to cloud storagedevices and locations or on any other repository that allows streamingimagery data, annotations and metadata, including semantic metadata andannotations, to be combined in native single file format structures forviewing medical images 13, including clinical data repositories,personalized clinical knowledge repositories, clinical cognitive vismemevaults and metadata repositories.

Streaming imagery data into local area networks 3 wherein TIMSClini-Docks 6 are connected directly to medical modalities 7, 8, 9acquiring live streaming imagery data or archived streaming imagerydata, streaming that imagery data to TIMS Clini-Ports 10 via local areanetworks 3. TIMS Clini-Ports 10 acquire lists 15 of available medicalmodalities 7, 8, 9 from a local area network 3. Included in this networkis are TIMS Clini-Pod Network Servers (CNS) 2 having associateddatabases, identifying each participant cognitive collaborant and thestreaming imagery data available to each participant cognitivecollaborant; identifying each participant cognitive collaborant thestreaming imagery data that is available on each participant cognitivecollaborant's computer. Also, local area networks 3 can be connected tothe internet 12.

When a participant cognitive collaborant wants to view medical imageryand collaborate on that streaming imagery data with others, thatparticipant cognitive collaborant selects a channel on the multi-channelsource selection tab for viewing streaming imagery data 15, 25 so he/shecan initiate a collaboration session, as depicted in FIG. 3. Whenparticipant cognitive collaborants are in a collaboration session, TIMSClini-Pod Network Servers (CNS) 2 are providing updates to eachparticipant cognitive collaborant's computer at a rapid frame rate soeach participant cognitive collaborant's computer concurrently displaysthe same imagery. In other words, TIMS Clini-Pod Network Servers (CNS) 2updates any changes to each and all of the streaming imagery data oneach of the participant cognitive collaborant's computers withsynchronized signals sent over local area networks 3 dynamically suchthat all streaming imagery data on all participant cognitive collaborantcomputer displays are the same, including sending each participantcognitive collaborant's input illustrations 18, which include,telestrations 21, drawings 22, and annotations 23, and illustrationsover the streaming imagery data 13 made by any of the participantcognitive collaborants 10.

TIMS Clini-Pod Network Servers (CNS) 2 with dynamic signalsynchronization ensures that the same imagery refresh rate isconcurrently available on all participant cognitive collaborantcomputers. TIMS Clini-Pod Network Servers (CNS) 2 use a process of localregistration to identify the image frames needed for viewing on each ofthe participant cognitive collaborant computers, and send to each ofthem only the image frames necessary for participation in acollaboration session. TIMS Clini-Pod Network Servers (CNS) 2 enableseach participant cognitive collaborant 10 to use a scalable window soall input illustrations 18 for each and every participant cognitivecollaborant 10 are dynamically ratio metric based on the underlyingimage aspect ratio of the respective computer of each participantcognitive collaborant 10. Each participant cognitive collaborant 10views what every other authorized participant cognitive collaborant 10views in that session.

TIMS Clini-Pod Network Servers (CNS) 2 distribute copies of streamingimagery data selected for use during a collaboration session to each ofthe participant cognitive collaborants. Since participant cognitivecollaborants 10 collaborate only with copies of images, they do notalter the original streaming imagery data in any way. TIMS Clini-PodNetwork Servers (CNS) 2 with dynamic signal synchronization allows atleast one participant cognitive collaborant 10 to telestrate 21, draw22, annotate 23, input illustrations 18 over the streaming imagery data13 in a concurrently collaboration session wherein a participantcognitive collaborant 10 is telestrating 21, drawing 22, annotating 23input illustrations 18 over the streaming imagery data 13. This approachof generating input illustrations 18 on TIMS Clini-Pod Network Servers(CNS) 2, and distributing only those input illustrations 18, and not theunderlying images to each participant cognitive collaborant 10,significantly improves operating performance and reduces image latencyand wait times.

TIMS Clini-Pod Network Servers (CNS) 2 manage input illustrations 18from all participant cognitive collaborants 10 in a concurrentlycollaborative environment with image streams which can include multiplestreams of streaming imagery data. TIMS Clini-Pod Network Servers (CNS)2 manage participant cognitive collaborant 10 input illustrations 18,which include telestrations 21, drawings 22, and annotations 23 as theyare appended to that imagery 13, and encapsulate and save thoseparticipant cognitive collaborant input illustrations 18, which includetelestrations 21, drawings 22 and annotations 23 together with streamingimagery data 13, and relevant imagery metadata, including appendedimagery metadata, from the collaboration session in single file formatstructures, known as collaborated imagery files.

TIMS Clini-Pod Network Servers (CNS) 2 ‘single file encapsulate andsave’ functionality encapsulates and saves collaborated imagery files insingle file format structures, as may be required or allowed bystandards for clinical documentation or medical records storage,including those as specified in the DICOM Standard. Users canencapsulate and save collaborated imagery files locally in medialibraries or image data repositories on their respective computerstorage devices, as depicted in FIG. 4, which contain all of the inputillustrations 18 from all participant cognitive collaborants 10. Userscan also encapsulate and save collaborated imagery files in image datarepositories on TIMS Clini-Pod Network Servers (CNS) 2, in image datarepositories on picture archiving and communications systems PACS 4, inother image data repositories compliant with standards for digitalimaging and communications in medicine DICOM, or on any other datarepository that allows streaming imagery data, annotations and metadata,including semantic metadata and annotations, to be combined in nativesingle file format structures, including in such locations as datacontainers and data catalogs, clinical data repositories, personalizedclinical knowledge repositories, clinical cognitive vismeme vaults andmetadata repositories, on premises, as well as on cloud storage devicesand locations.

TIMS Clini-Pod Network Servers (CNS) 2 create session logs that includecollaboration session identification, participant cognitive collaborantinformation, information about streaming imagery data, includingassociated patient metadata, along with session dates and times, asshown in FIG. 9.

In one embodiment, several participant cognitive collaborants 10, alsoknown as Radiologist, Pathologist and Oncology Surgeon, utilize thenetwork system 1 to collaborate in the provision of oncology care.

At Time 1, Radiologist retrieves patient's archived medical imagery froma PACS 4 image repository. Radiologist detects a suspicious nodule onseveral images and inputs telestrations 21 and drawings 22 indicatingthe location of the nodule, along with text annotations 23 characteringits clinical significance and voice annotations 23 summarizing hisfindings. Radiologist utilizes the ‘single file encapsulate and save’functionality of the network system 1 to incorporate those inputillustrations 18, together with medical imagery data 13 and identifyingpatient metadata, in single file format structures, known as acollaborated imagery file (CIF #1). Radiologist archives the CIF #1,which has been encapsulated and saved in a single file format compliantwith the DICOM Standard, and sends to PACS 4 for review and discussionwith other members of the oncology care team.

At Time 2, Radiologist invites Pathologist to a collaboration session todiscuss his findings of a suspicious nodule as described in CIF #1.While both participant cognitive collaborants 10 are concurrentlyviewing CIF #1, Radiologist retrieves several additional collaboratedimagery files from his local media library, and from PACS 4, of relevantprior patient medical imagery for display and viewing during thecollaboration session, as shown in FIG. 4. Participant cognitivecollaborants 10 record, encapsulate and save their input illustrations18 for each of several imagery files selected for discussion during thecollaboration session, as CIF #2, #3, #4. Pathologist combines CIF #1with CIF #2, #3, #4 as collaborated imagery study (CIS #1) and storesCIS #1 on PACS 4 for subsequent review and discussion with OncologySurgeon, who was unavailable at Time 2 to join collaboration session.

At Time 3, Oncology Surgeon reviews CIS #1 and selects CIF #4 to createa surgical roadmap to guide tumor excision using input illustrations 18,which include telestrations 21, drawings 22, and voice annotations 23.Oncology Surgeon saves surgical roadmap as CIF #5.

At Time 4, Oncology Surgeon retrieves surgical roadmap (CIF #5), forintra-operative guidance during tumor removal.

At Time 5, during surgery, Oncology Surgeon invites Radiologist andPathologist for intra-operative consultation during tumor excision.

At Time 6, participant cognitive collaborants—Oncology Surgeon,Radiologist, and Pathologist—utilize network system 1 to retrieve andconcurrently view nodule (CIF #1), tumor pathology images (CIF #2, #3,#4), and surgical roadmap (CIF #5) from PACS 4, along with livestreaming imagery data from endoscope 13 used during tumor excision.

Periodically during the surgical procedure, at Times 7, 8, 9, OncologySurgeon consults with Pathologist to confirm sufficiency of marginsaround excised tumor. Pathologist confirms sufficiency of margins withtelestrations 21, drawings 22, and text annotations 23, over liveendoscopy images, saving all those input illustrations 18, together withassociated streaming imagery data 13 in single file format structure asCIF #6.

At Time 10, Oncology Surgeon retrieves CIF #6 from PACS 4, whichcontains Pathologist's input illustrations 18 regarding excised tumormargins, and dictates a post-operative surgical report adding voiceannotations 23, to telestrations 21, and drawings 22 to endoscopicimages from excision surgery and saving in single file format structureas CIF #7.

At Time 11, Oncology Surgeon combines pre-operative surgical roadmap CIF#5 with post-operative surgical report CIF #7, along with pre-operativeimage study CIS #1 (which includes CIF #1, #2, #3, #4) intocomprehensive clinical report (CIS #2) for distribution to the oncologycare team.

Oncology Surgeon can encapsulate and save CIS #2 in single file formatstructures as specified in the DICOM Standard and send to PACS 4.Oncology Surgeon utilizes the ‘single file encapsulate and save’functionality of the network system to encapsulate and save CIS #2 insingle file format structures as specified in the DICOM Standard andsend to PACS 4. Oncology Surgeon can also encapsulate and save CIS #2 insingle file format structures as may be required or allowed for clinicaldocuments, for storage in patient's electronic medical record, or forpatient billing.

At Time 12, Oncology Surgeon retrieves CIS #2 from PACS 4, utilizes thenetwork system 1 to remove all relevant identifying patient metadata,and encapsulates and saves as an anonymized collaborated imagery study(CIS #3) for use as teaching files with surgical fellows.

In another embodiment, a participant cognitive collaborant 10, known asHospitalist, remotely monitors live streaming imagery data 13 from asurgical procedure in an operating room on channel one, and archivedstreaming imagery data 13 of a patient recovering in Intensive CareUnit, on channel two. While monitoring streaming imagery data 13 onchannels one and two, as depicted in FIG. 3 and FIG. 7, Hospitalistaccepts an invitation to join a collaboration session on channel threeto monitor and consult live on a diagnostic procedure in the emergencyroom, as shown in FIG. 6. The live consultation involves review ofpatient images from an analog ultrasound machine and a digital CTscanner in the emergency room. During the collaboration session in theemergency room on channel three, Hospitalist utilizes the multi-channelviewing capability of Applicant's network system 1 to continue livemonitoring of streaming imagery data 13 on channel one and channel two,and to retrieve and view additional archived imagery data 13 of patientrecovery in Intensive Care Unit.

In another embodiment, a patient is recalled to undergo a second PET/MRIscan. The previous test yielded inconclusive, due to patient motionduring image capture, thus requiring a costly retest. During the secondtest, Radiologist was able to review the MRI images captured 13 duringthe first portion of the test, while the patient was still being imagedin PET unit and confirm that the second MRI scan was useable.Radiologist was able to advise Attending Molecular Pathologist duringPET scan 13 of additional regions of interest with input illustrations18 for further investigation.

In another embodiment, Oncologist wishes to convene a virtual tumorboard for the following day involving multi-specialist collaborationwith a patient's Radiologist, Pathologist, Oncology Surgeon and himselfOncologist sends invitations to colleagues along with severalcollaborated imagery files he wishes to review during the collaborationsession. Radiologist and Pathologist confirm availability, but OncologySurgeon is unable to attend. However, Oncology Surgeon is able toannotate 23 with telestrations 21 and drawings 22 on several key images13 included in the collaborated imagery study sent with the sessioninvitation. Oncology Surgeon also includes his clinical notes and anaudio file along with his report, together all encapsulated as a CIF andreturned to the session host.

During the collaboration session the following day, the host Oncologistretrieves patient images from PACS 4 and from his local media library 25containing the CIF 13, 18 sent to him from Oncology Surgeon, viewingboth images concurrently when colleagues from radiology and pathologyjoin the collaboration session. During the collaboration session,Pathologist is monitoring on the third channel of the multi-channelstreamer 7, 8, 9, 25, a tumor removal of another patient in theoperating room, advising that Oncology Surgeon intra-operativelyregarding sufficiency of margins of tumor removal from that patient.Oncology Surgeon is able to share live imagery 13 of the tumor removalwith the radiology and oncology colleagues who have joined the virtualtumor board collaboration session.

At the conclusion of the collaboration session, host Oncologistencapsulates and saves input illustrations 18 from participant cognitivecollaborants 10, including encapsulated audio clinical notes and biopsyreports as clinical documents, saving them as collaborated imagery filesand sending them to all participant cognitive collaborants 10 as well asinvitees unable to attend. Additionally, the CIFs 13, 18 are sent toPACS 4 for inclusion in the patient's electronic medical records as wellto patient's referring clinician.

Other embodiments of the invention include applications for cognitivevalue creation with knowledge mapping, advanced and augmented dataanalytics, as depicted in FIGS. 19 through 24; for augmenting clinicalintelligence with semantic metadata and imagery annotation, as depictedin FIGS. 25 through 31; for cognitively-enriched enterprise imaging withstreaming imagery informatics, as depicted in FIGS. 33 through 38; forcollaborative precision medicine with multiomic data analytics, asdepicted in FIGS. 39 through 50; for informatics-enriched imagery guidedintervention, including robotic-assisted surgery, as depicted in FIGS.51 through 62; for machine learning with medical imaging, including deeplearning, transfer learning, reinforcement learning, convolutionalneural networks, recurrent neural networks, LSTM and NLP, as depicted inFIGS. 63 through 75; for precision guided biomedical nanorobotics, asdepicted in FIGS. 76 through 79; and for personalized precision targetedtheranostic nanomedicine, as depicted in FIGS. 80 through 85.

Various techniques for machine learning with medical imaging arespecified in FIG. 63, including among others, artificial neuralnetworks, ensemble learning and multiple instance learning. Otherapplications for machine learning in medicine are depicted in FIG. 74, aPeriodic Table of Artificial Intelligence with “Elementary” PAIRTechniques [Perceive-Assess-Infer-Respond]. Those AI applicationsinclude speech, audio and image recognition; data analytics, inferenceand reasoning; text extraction, problem solving and decision making;language understanding and generation; knowledge refinement, categoryand relationship learning [semantics]; as well as communications,manipulation and control.

Other embodiments of the invention may include, but are not limited to,various combinations of algorithms, applications, tools and techniquesfor machine learning in medicine, e.g., deep learning, transferlearning, reinforcement learning, convolutional neural networks,recurrent neural networks, LSTM networks, natural language processingand augmented analytics, as well as those specified above.

The principle preferred embodiments and modes of operation of thepresent invention have been described in the forgoing specification. Theinvention which is intended to be protected herein, however, is not tobe construed as limited to the particular embodiments disclosed, sincethese embodiments are to be regarded as illustrative rather thanrestrictive. Variations and changes may be made by others withoutdeparting from the spirit of this invention. Accordingly, it isexpressly intended that all such variation and changes which fall withinthe spirit and scope of the claims be embraced thereby.

What is claimed is:
 1. A network system enabling multichannelmultiplexed communications, collaboration, consultation and instruction,as well as recursive cognitive enrichment and collaborative knowledgeexchange, with streaming imagery data during collaboration sessions,practiced by and among at least one or more participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, the network system enabling both synchronous and asynchronousmultimodal clinical communications, collaboration, consultation andinstruction, including recursive cognitive enrichment and collaborativeknowledge exchange, with streaming imagery data during various stages ofmedical disease and injury management, including detection, diagnosis,prognosis, treatment, measurement, monitoring and reporting, as well asworkflow optimization with operational analytics for outcomes,performance, results, resource utilization, resource consumption andcosts, allowing each participant cognitive collaborant to capture,retrieve and concurrently view at least one source of streaming medicalmodality imagery data, and at least one or more sources of heterogeneousstreaming imagery data, medical and non-medical streaming imagery data,and combinations thereof including images, video, modality imagery,audio, video and haptic wave forms and files, multiomic—phenotypic,genomic, metabolomic, pathomic, radiomic, radiopathomic andradiogenomic—maps and clinical data sets, biometric maps and movies,hapmaps, heat maps, data stream visualizations, structured reports,interactive media reports, clinical documents and key performanceindicators, both live and archived streaming imagery data, enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, withstreaming imagery data in collaboration sessions practiced by and amongat least one or more participant cognitive collaborants during variousstages of medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs, each participant cognitive collaborant able toview, curate, annotate and tag the heterogeneous streaming imagery data,comprising a tele-visual imagery informatics management systemincluding, at least one or more tele-visual imagery informaticsmanagement system clini-docks, wherein each clini-dock is adapted forindependent acquisition and transmission of signals from other sourcesof streaming imagery data at native, enhanced or reduced resolutions andnative enhanced or reduced frame rates, used for the acquisition andtransmission of, live or archived streaming imagery data, includingimages, video, modality imagery, audio, video and haptic wave forms andfiles, multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators, analog or digital video signals in standardor non-standard resolutions, medical or non-medical imagery, incompressed or uncompressed imagery formats; at least one or moretele-visual imagery informatics management system clini-pod networkservers, wherein each server is a neurosynaptic network node comprisingat least one streamer, splitter, router, server and storage deviceenabling at least one or more participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, toconcurrently view, communicate, collaborate, consult and instruct amongparticipant cognitive collaborants, including curation, annotation andtagging, using at least one or more sources of streaming imagery dataacquired and transmitted by tele-visual imagery informatics managementsystem clini-docks, including live streaming imagery data, archivedstreaming imagery data, appended streaming imagery metadata, includingappended semantic metadata and annotations, cognitive collaborantannotations, and archived collaborated imagery files during asynchronous or asynchronous collaboration session, establishing andmaintaining channel communications for each and all of the sources ofstreaming imagery data for at least one or more participant cognitivecollaborant during a collaboration session, enabling at least one ormore participant cognitive collaborants in at least one or morelocations, to concurrently view, communicate, collaborate, consult andinstruct among participant cognitive collaborants using at least one ormore sources of live streaming imagery data, archived streaming imagerydata, appended streaming imagery metadata, cognitive collaborantannotations, and archived collaborated imagery files, includingcuration, annotation and tagging from each participant cognitivecollaborant during a collaboration session, managing and controlling atleast one or more associated databases, and privileges forauthorization, authentication, identity management, security, access,publication and distribution for viewing, communicating, collaborating,consulting and instructing among participant cognitive collaborants,including managing and controlling security tokens providing access forcognitive collaborants maintained in security metadata repositories,blockchain metadata repositories and blockchain data ledgers, managingand controlling privileges for at least one or more participantcognitive collaborants to view, curate, annotate, tag, encapsulate,save, store, retrieve and distribute live streaming imagery data,archived streaming imagery data, appended streaming imagery metadata,including appended semantic metadata and annotations, cognitivecollaborant annotations, and archived collaborated imagery files foreach participant cognitive collaborant during collaboration sessions,enabling both synchronous and asynchronous bidirectional communicationswith combinations of at least one or more local area networks, at leastone or more wide area networks, including internet, and at least one ormore streaming imagery data repositories during at least one or morecollaboration sessions, enabling identification, tracking and monitoringof participant cognitive collaborants by assignment of unique colors forannotations of streaming imagery data, archived collaborated imageryfiles and cognitive collaborant annotations, including telestrations,drawings, illustrations, alpha-numeric text annotations, imageannotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, haptic annotations, document annotations, outcomesannotations, performance annotations, results annotations, resourceconsumption annotations, resource utilization annotations and costsannotations, enabling colorized telestration, annotation and masking ofcolorized attention maps and colorized prediction bases for explainableartificial intelligence by participant cognitive collaborants byassignment of unique colors for annotations of streaming imagery data,archived collaborated imagery files and cognitive collaborantannotations, including telestrations, drawings, illustrations,alpha-numeric text annotations, image annotations, wave formannotations, voice annotations, video annotations, augmented realityimagery annotations, 3D/4D imagery annotations, haptic annotations,document annotations, outcomes annotations, performance annotations,results annotations, resource consumption annotations, to resourceutilization annotations and costs annotations; and at least one or moretele-visual imagery informatics management system clini-ports allowingat least one or more participant cognitive collaborants, each, capturinglive streaming imagery data, capturing associated live streaming imagerymetadata, including semantic metadata and annotations, retrievingarchived streaming imagery data, retrieving archived associated imagerymetadata, including archived semantic metadata and annotations, andtransporting live streaming imagery data, transporting associated livestreaming imagery metadata, including semantic metadata and annotations,and transporting live streaming imagery data, associated live streamingmetadata, including semantic metadata and annotations, archivedstreaming imagery data, associated archived streaming metadata,including archived semantic metadata and annotations, into collaborationsessions, concurrently viewing, communicating, collaborating, consultingand instructing among participant cognitive collaborants using at leastone or more sources of streaming imagery data, curating, annotating andtagging streaming imagery data, including telestrations, drawings,illustrations, alpha-numeric text annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, haptic annotations, document annotations, outcomesannotations, performance annotations, results annotations, resourceconsumption annotations, resource utilization annotations and costsannotations, and encapsulating streaming imagery data and associatedstreaming imagery metadata, including semantic metadata and annotations,together with cognitive collaborant annotations in native, single fileformat structures, and saving said streaming imagery data and saidassociated streaming imagery metadata, including semantic metadata andannotations, together with said cognitive collaborant annotations in atleast one or more collaborated imagery files during collaborationsessions, including asynchronous or synchronous collaborations with atleast one or more participant cognitive collaborants, communicating,collaborating, consulting and instructing, including viewing, curating,annotating and tagging, using at least one or more sources of streamingimagery data shared among at least one or more participant cognitivecollaborants with a multi-channel stream viewer that enables capture,retrieval and concurrent viewing of both live and archived medicalimagery streams together with associated metadata, including semanticmetadata and annotations, during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs, independently addingsources of streaming imagery data, adjust, enhance or reduce resolutionsor frame rates of streaming imagery data with a multi-channelcommunications control interface, and independently view thoseadditional channels of streaming imagery data and independently selectwhich of those channels to bring into a collaboration session, conveyinginstructions with two way communications among participant cognitivecollaborants, including source channel selection, for viewing, curating,annotating and tagging imagery data streams with telestrations,drawings, illustrations, alpha-numeric text annotations, imageannotations, wave form annotations, voice annotations, imageannotations, wave form annotations, video annotations, augmented realityimagery annotations, 3D/4D imagery annotations, haptic annotations,document annotations, outcomes annotations, performance annotations,results annotations, resource consumption annotations, resourceutilization annotations and costs annotations, and not reliant upon anyexternal communications network.
 2. The network system of claim 1 forthe acquisition and transmission of heterogeneous sources of streamingimagery data, enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instruction, aswell as recursive cognitive enrichment and collaborative knowledgeexchange, with streaming imagery data during collaboration sessions,practiced by and among at least one or more participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, each participant cognitive collaborant able to view, curate,annotate and tag the heterogeneous streaming imagery data, includingmedical video, medical modality imagery, medical wave form imagery, andclinical documents during various stages of medical disease and injurymanagement, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs, encapsulating and savingcollaborated annotations and tags together with streaming imagery data,relevant imagery metadata, including semantic metadata and annotations,including appended imagery metadata and appended semantic metadata andannotations, from the collaboration session in native, single fileformat structures, known as collaborated imagery files, storingcollaborated imagery files from all participant cognitive collaborantslocally in media libraries or image data repositories on theirrespective computer storage devices, in image data repositories ontele-visual imagery informatics management system servers, in image datarepositories on cloud storage devices and locations, in image datarepositories on picture archiving and communications systems, in otherimage data repositories compliant with standards for digital imaging andcommunications in medicine, or in any other data repository that allowsstreaming imagery data, annotations and metadata, including semanticmetadata and annotations, to be combined in native single file formatstructures, including in such locations as data containers and datacatalogs, clinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations, retrieving collaborated imagery files from all participantcognitive collaborants stored locally in media libraries or image datarepositories on their respective computer storage devices, in image datarepositories on tele-visual imagery informatics management systemservers, in image data repositories on cloud storage devices andlocations, in image data repositories on picture archiving andcommunications systems, in other image data repositories compliant withstandards for digital imaging and communications in medicine, or in anyother data repository that allows streaming imagery data, annotationsand metadata, including semantic metadata and annotations, to becombined in native single file format structures, including in suchlocations as data containers and data catalogs, clinical datarepositories, personalized clinical knowledge repositories, clinicalcognitive vismeme vaults and metadata repositories, on premises, as wellas on cloud storage devices and locations, publishing and distributingcollaborated imagery files in known native, single file formatstructures, including those used for digital imaging and communicationsin medicine comprising both core and non-core data element tags,together with conformance statements that enable prior evaluation andtesting of streaming imagery equipment components without an actualphysical connection, all of which facilitate network connectivity forimagery equipment components, communication interoperability for imagerydata systems, and exchange of collaborated imagery files.
 3. The networksystem of claim 1 for the acquisition and transmission of medicalstreaming imagery data, including medical images, medical video, medicalmodality imagery, medical wave form imagery, clinical maps,multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators, the network system preserving the clinicalintegrity of medical streaming imagery data from medical devices,systems and equipment cleared for medical use, including clinicaldiagnostic purposes, care delivery and patient monitoring, enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, withstreaming imagery data during collaboration sessions, practiced by andamong at least one or more participant cognitive collaborants, includingpersons, machines, devices, neural networks, robots and algorithms, andheterogeneous networked teams composed thereof, including viewing,curating, annotating and tagging streaming medical imagery data duringvarious stages of medical disease and injury management, includingdetection, diagnosis, prognosis, treatment, measurement, monitoring andreporting, as well as workflow optimization with operational analyticsfor outcomes, performance, results, resource utilization, resourceconsumption and costs, including recursive cognitive enrichmentsthereof, for use with medical devices, equipment, systems, image anddata repositories, in native, single file format structures, includingthose compliant with standards for digital imaging and communications inmedicine.
 4. The network system of claim 1 for enabling both synchronousand asynchronous multimodal clinical communications, collaboration,consultation and instruction, as well as recursive cognitive enrichmentand collaborative knowledge exchange, with streaming imagery data duringcollaboration sessions, among participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, togetherwith collaborated imagery files created from cognitive collaborantannotations, session metadata and medical streaming imagery data duringcollaboration sessions, practiced by and among at least one or moreparticipant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, including data cleared for clinicaldiagnostic purposes, that can be viewed, curated, annotated, tagged,encapsulated and saved together as collaborated medical imagery filesand cleared for use with approved medical devices, equipment, systems,image and data repositories, in native, single file format structures,including those compliant with standards for digital imaging andcommunications in medicine.
 5. The network system of claim 1 forarchived collaborated imagery files that can be retrieved for usetogether with streaming imagery data during synchronous or asynchronouscollaboration sessions, revised, appended, viewed, curated, annotated,tagged, encapsulated and saved in native, single file format structures,including those compliant with standards for digital imaging andcommunications in medicine, during collaboration sessions practiced byand among at least one or more participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, and madeavailable for use together with streaming imagery data during current orsubsequent collaboration sessions.
 6. A method enabling multichannelmultiplexed communications, collaboration, consultation and instruction,as well as recursive cognitive enrichment and collaborative knowledgeexchange, with streaming imagery data practiced by and among at leastone or more participant cognitive collaborants, including persons,machines, devices, neural networks, robots and algorithms, andheterogeneous networked teams composed thereof, the network systemenabling both synchronous and asynchronous multimodal clinicalcommunications, collaboration, consultation and instruction, as well asrecursive cognitive enrichment and collaborative knowledge exchange,with streaming imagery data during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs, allowing each participantcognitive collaborant to capture, retrieve and concurrently view atleast one source of streaming medical modality imagery data, and atleast one or more sources of heterogeneous streaming imagery data,medical and non-medical streaming imagery data, and combinations thereofincluding images, video, modality imagery, audio, video and haptic waveforms and files, multiomic—phenotypic, genomic, metabolomic, pathomic,radiomic, radiopathomic and radiogenomic—maps and clinical data sets,biometric maps and movies, hapmaps, heat maps, data streamvisualizations, structured reports, interactive media reports, clinicaldocuments and key performance indicators, both live and archivedstreaming imagery data, enabling both synchronous and asynchronousmultimodal clinical communications, collaboration, consultation andinstruction, as well as recursive cognitive enrichment and collaborativeknowledge exchange, with streaming imagery data in collaborationsessions practiced by and among at least one or more participantcognitive collaborants during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs, each participant cognitivecollaborant able to view, curate, annotate and tag the heterogeneousstreaming imagery data, comprising a tele-visual imagery informaticsmanagement system consisting of the following essential components: atleast one or more tele-visual imagery informatics management systemclini-docks, wherein each clini-dock is adapted for independentacquisition and transmission of signals from other sources of streamingimagery data at native, enhanced or reduced resolutions and native,enhanced or reduced frame rates, used for the acquisition andtransmission of, live or archived streaming imagery data, includingimages, video, modality imagery, audio, video and haptic wave forms andfiles, multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators, analog or digital video signals in standardor non-standard resolutions, medical or non-medical imagery, incompressed or uncompressed imagery formats; at least one or moretele-visual imagery informatics management system clini-pod CNS networkservers, wherein each server is a neurosynaptic network node comprisingat least one streamer, to splitter, router, server and storage devicethat enables at least one or more participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, toconcurrently view, communicate, collaborate, consult and instruct amongparticipant cognitive collaborants using at least one or more sources ofstreaming imagery data acquired and transmitted by tele-visual imageryinformatics management system clini-docks, including live streamingimagery data, archived streaming imagery data, appended streamingimagery metadata, including appended semantic metadata and annotations,cognitive collaborant annotations, and archived collaborated imageryfiles during a synchronous or asynchronous collaboration session,enables concurrent collaboration including viewing, curation, annotationand tagging with each and all of the one or more sources of streamingimagery data acquired and transmitted by tele-visual imagery informaticsmanagement system clini-docks, establishes and maintains channelcommunications for each and all of the one or more sources of streamingimagery data each participant cognitive collaborant wishes to view,monitor and collaborate with, enables at least one or more participantcognitive collaborants to concurrently view, communicate, collaborate,consult and instruct, including curation, annotation and tagging, withlive streaming imagery data, archived imagery data, appended imagerymetadata, including appended semantic metadata and annotations,collaborated annotations, and archived collaborated imagery files duringa synchronous or asynchronous collaboration session, enables at leastone or more participant cognitive collaborant in multiple locations,some of whom may be located remotely to the sources of streaming imagerydata, to concurrently view, communicate, collaborate, consult andinstruct, including curation, annotation and tagging, with livestreaming imagery data, archived imagery data, appended imagerymetadata, including appended semantic metadata and annotations,collaborated annotations, and archived collaborated imagery files fromeach participant cognitive collaborant during the collaboration session,dynamically manages and controls with at least one or more associateddatabases, authorization, authentication, identity management, security,and access, publication and distribution privileges for viewing,communicating, collaborating, consulting and instructing, and cognitivecollaborant privileges, including curation, annotation, tagging,encapsulation, saving, storage, retrieval and distribution of livestreaming imagery data, archived imagery data, appended imagerymetadata, including appended semantic metadata and annotations,collaborated annotations, and archived collaborated imagery files foreach participant cognitive collaborant during collaboration sessions,including managing and controlling security tokens providing access forcognitive collaborants maintained in security metadata repositories,blockchain metadata repositories and blockchain data ledgers, enablesboth synchronous and asynchronous bidirectional communications with atleast one or more local area networks, at least one or more wide areanetworks (internet) including imagery data repositories and combinationsthereof during multiple collaboration sessions, enables identification,tracking and monitoring of participant cognitive collaborants byassignment of unique colors for annotations of streaming imagery data,archived collaborated imagery files and cognitive collaborantannotations, that include telestrations, drawings, illustrations,alpha-numeric text annotations, as well as cognitive collaborantannotations combined with alpha-numeric text annotations, imageannotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, haptic annotations, document annotations, outcomesannotations, performance annotations, results annotations, resourceconsumption annotations, resource utilization annotations and costsannotations, enables colorized telestration, annotation and masking ofcolorized attention maps and colorized prediction bases for explainableartificial intelligence by participant cognitive collaborants byassignment of unique colors for annotations of streaming imagery data,archived collaborated imagery files and cognitive collaborantannotations, including telestrations, drawings, illustrations,alpha-numeric text annotations, image annotations, wave formannotations, voice annotations, video annotations, augmented realityimagery annotations, 3D/4D imagery annotations, haptic annotations,document annotations, outcomes annotations, performance annotations,results annotations, costs annotations, resource consumption annotationsand resource utilization annotations; and at least one or moretele-visual imagery informatics management system clini-ports thatallows for multiple participant cognitive collaborants, each of whom cancapture live streaming imagery data together with associated imagerymetadata, including semantic metadata and annotations, and bring intothe collaboration session, retrieve archived streaming imagery datatogether with associated imagery metadata, including semantic metadataand annotations, and bring into the collaboration session, concurrentlyview, communicate, collaborate, consult and instruct with streamingimagery data, enables curation, annotation and tagging that streamingimagery data with collaborated annotations that include telestrations,drawings, illustrations, alpha-numeric text annotations, imageannotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, haptic annotations, document annotations, outcomesannotations, performance annotations, results annotations, resourceconsumption annotations, resource utilization annotations and costsannotations, enables encapsulation and saving collaborated streamingimagery data and archived imagery metadata, including archived semanticmetadata and annotations, together with appended imagery metadata,including appended semantic metadata and annotations, and collaboratedannotations and from each collaboration session, including asynchronousor synchronous collaboration with at least one or more participantcognitive collaborants, in native, single file format structures, knownas collaborated imagery files, enables multimodal clinicalcommunications, collaboration, consultation and instruction, includingviewing, curating, annotating and tagging, using at least one or moresources of streaming imagery data shared among at least one or moreparticipant cognitive collaborants with a multi-channel stream viewerthat enables capture, retrieval and concurrent viewing of both live andarchived medical imagery streams together with associated metadataduring various stages of medical disease and injury management,including detection, diagnosis, prognosis, treatment, measurement,monitoring and reporting, as well as workflow optimization withoperational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs, enables independentlyadding sources of streaming imagery data, adjust, enhance or reduceresolutions or frame rates of streaming imagery data with amulti-channel communications control interface, and independently viewthose additional channels of streaming imagery data and independentlyselect which of those channels to bring into a collaboration session,enables conveying instructions with two way communications amongparticipant cognitive collaborants, including source channel selection,for viewing, curating, annotating and tagging imagery data streams withtelestrations, drawings, illustrations, alpha-numeric text annotations,image annotations, wave form annotations, voice annotations, videoannotations, augmented reality imagery annotations, 3D/4D imageryannotations, haptic annotations, document annotations, outcomesannotations, performance annotations, results annotations, resourceconsumption annotations, resource utilization annotations and costsannotations, and not reliant upon any external communications network.7. The method of claim 6 for the acquisition and transmission ofheterogeneous sources of streaming imagery data, enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, duringcollaboration sessions practiced by and among at least one or moreparticipant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, persons, each participant cognitivecollaborant able to view, curate, annotate and tag the heterogeneousstreaming imagery data, including medical video, medical modalityimagery, medical wave form imagery, and clinical documents duringvarious stages of medical disease and injury management, includingdetection, diagnosis, prognosis, treatment, measurement, monitoring andreporting, as well as workflow optimization with operational analyticsfor outcomes, performance, results, resource utilization, resourceconsumption and costs, as well as encapsulate and save collaboratedannotations and tags together with the heterogeneous streaming imagerydata, including medical video, medical modality imagery, medical waveform imagery, and clinical documents, and save collaborated annotationstogether with streaming imagery data, relevant imagery metadata,including semantic metadata and annotations, and appended imagerymetadata, including appended semantic metadata and annotations, from thecollaboration session in native, single file format structures, known ascollaborated imagery files; storing collaborated imagery files from allparticipant cognitive collaborants locally in media libraries or imagedata repositories on their respective computer storage devices, in imagedata repositories on tele-visual imagery informatics management systemservers, in image data repositories on cloud storage devices andlocations, in image data repositories on picture archiving andcommunications systems, in other image data repositories compliant withstandards for digital imaging and communications in medicine, or in anyother data repository that allows streaming imagery data, annotationsand metadata, including semantic metadata and annotations, to becombined in native single file format structures, including in suchlocations as data containers and data catalogs, clinical datarepositories, personalized clinical knowledge repositories, clinicalcognitive vismeme vaults and metadata repositories, on premises, as wellas on cloud storage devices and locations, retrieving collaboratedimagery files from all participant cognitive collaborants stored locallyin media libraries or image data repositories on their respectivecomputer storage devices, in image data repositories on tele-visualimagery informatics management system servers, in image datarepositories on cloud storage devices and locations, in image datarepositories on picture archiving and communications systems, in otherimage data repositories compliant with standards for digital imaging andcommunications in medicine, or in any other data repository that allowsstreaming imagery data, annotations and metadata, including semanticmetadata and annotations, to be combined in native single file formatstructures, including in such locations as data containers and datacatalogs, clinical data repositories, personalized clinical knowledgerepositories, clinical cognitive vismeme vaults and metadatarepositories, on premises, as well as on cloud storage devices andlocations, publishing and distributing collaborated imagery files inknown native, single file format structures, including those used fordigital imaging and communications in medicine comprising both core andnon-core data element tags, together with conformance statements thatenable prior evaluation and testing of streaming imagery equipmentcomponents without an actual physical connection, all of whichfacilitate network connectivity for imagery equipment components,communication interoperability for imagery data systems, and exchange ofcollaborated imagery files.
 8. The method of claim 6 for the acquisitionand transmission of medical streaming imagery data, including medicalimages, medical video, medical modality imagery, medical wave formimagery, clinical maps, multiomic—phenotypic, genomic, metabolomic,pathomic, radiomic, radiopathomic and radiogenomic—maps and clinicaldata sets, biometric maps and movies, hapmaps, heat maps, data streamvisualizations, structured reports, interactive media reports, clinicaldocuments and key performance indicators, the method preserving theclinical integrity of medical streaming imagery data from medicaldevices systems and equipment cleared for medical use, includingclinical diagnostic purposes, care delivery and patient monitoring,enabling both synchronous and asynchronous multimodal clinicalcommunications, collaboration, consultation, instruction, as well asrecursive cognitive enrichment and collaborative knowledge exchange,practiced by and among at least one or more participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, including viewing, curation, annotation and tagging ofstreaming medical imagery data during various stages of medical diseaseand injury management, including detection, diagnosis, prognosis,treatment, measurement, monitoring and reporting, as well as workflowoptimization with operational analytics for outcomes, performance,results, resource utilization, resource consumption and costs, includingrecursive cognitive enrichments thereof, for use with medical devices,equipment, systems, image and data repositories, in native, single fileformat structures, including those compliant with standards for digitalimaging and communications in medicine.
 9. The method of claim 6 forenabling both synchronous and asynchronous multimodal clinicalcommunications, collaboration, consultation and instruction, as well asrecursive cognitive enrichment and collaborative knowledge exchange,with streaming imagery data in collaboration sessions among participantcognitive collaborants, including persons, machines, devices, neuralnetworks, robots and algorithms, and heterogeneous networked teamscomposed thereof, together with collaborated imagery files created fromcognitive collaborant annotations, session metadata and medicalstreaming imagery data during collaboration sessions, practiced by andamong at least one or more participant cognitive collaborants, includingpersons, machines, devices, neural networks, robots and algorithms, andheterogeneous networked teams composed thereof, including data clearedfor clinical diagnostic purposes, that can be viewed, curated,annotated, tagged, encapsulated and saved together as collaboratedmedical imagery files and cleared for use with approved medical devices,equipment, systems, image and data repositories, including thosecompliant with standards for digital imaging and communications inmedicine.
 10. The method of claim 6 for archived collaborated imageryfiles that can be retrieved for use together with streaming imagery dataduring synchronous or asynchronous collaboration sessions, revised,appended, viewed, curated, annotated, tagged, encapsulated and saved innative, single file format structures, including those compliant withstandards for digital imaging and communications in medicine, duringcollaboration sessions, practiced by and among at least one or moreparticipant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, and made available for use togetherwith streaming imagery data during current or subsequent collaborationsessions.
 11. The method of claim 6 adapted for recursive cognitiveenrichment and collaborative mind-machine knowledge exchange between andamong participant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, with neurosynaptic networkconnectivity enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instruction,that includes viewing, curating, annotating and tagging, using at leastone or more sources of multichannel, multiplexed heterogeneous streamingimagery data, including both medical and non-medical streaming imagerydata, and combinations thereof, and together with images, video,modality imagery, waveforms, audio and haptic files,multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs.
 12. The method of claim 6adapted for informatics-enriched learning, specialist skills acquisitionand accelerated knowledge exchange with multimodal clinical instructionby and among participant cognitive collaborants, including persons,machines, devices, neural networks, robots and algorithms, andheterogeneous networked teams composed thereof, with neurosynapticnetwork connectivity enabling both synchronous and asynchronousmultimodal clinical communications, collaboration, consultation andinstruction, that includes viewing, curating, annotating and tagging,using at least one or more sources of multichannel, multiplexedheterogeneous streaming imagery data, including both medical andnon-medical streaming imagery data, and combinations thereof, andtogether with images, video, modality imagery, waveforms, audio andhaptic files, multiomic—phenotypic, genomic, metabolomic, pathomic,radiomic, radiopathomic and radiogenomic—maps and clinical data sets,biometric maps and movies, hapmaps, heat maps, data streamvisualizations, structured reports, interactive media reports, clinicaldocuments and key performance indicators during various stages ofmedical disease and injury management, including detection, diagnosis,prognosis, treatment, measurement, monitoring and reporting, as well asworkflow optimization with operational analytics for outcomes,performance, results, resource utilization, resource consumption andcosts.
 13. The method of claim 6 adapted for cognitively-enrichedenterprise imaging with streaming imagery informatics between and amongparticipant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, with neurosynaptic networkconnectivity enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instruction, aswell as recursive cognitive enrichment and collaborative knowledgeexchange, that includes viewing, curating, annotating and tagging, usingat least one or more sources of multichannel, multiplexed heterogeneousstreaming imagery data, including both medical and non-medical streamingimagery data, and combinations thereof, and together with images, video,modality imagery, waveforms, audio and haptic files,multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs.
 14. The method of claim 6adapted for collaborative precision medicine with multiomic dataanalytics between and among participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, withneurosynaptic network connectivity enabling both synchronous andasynchronous multimodal clinical communications, collaboration,consultation and instruction, as well as recursive cognitive enrichmentand collaborative knowledge exchange, that includes viewing, curating,annotating and tagging, using at least one or more sources ofmultichannel, multiplexed heterogeneous streaming imagery data,including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.
 15. The method of claim 6 adapted forinformatics-enriched imagery guided intervention, includingrobotic-assisted surgery, between and among participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, with neurosynaptic network connectivity enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, that includesviewing, curating, annotating and tagging, using at least one or moresources of multichannel, multiplexed heterogeneous streaming imagerydata, including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.
 16. The method of claim 6 adapted for machinelearning with medical imaging, including deep learning, transferlearning, reinforcement learning, convolutional neural networks,recurrent neural networks, long short term memory networks and naturallanguage processing, between and among participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, with neurosynaptic network connectivity enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, that includesviewing, curating, annotating and tagging, using at least one or moresources of multichannel, multiplexed heterogeneous streaming imagerydata, including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.
 17. The method of claim 6 adapted for precisionguided biomedical nanorobotics between and among participant cognitivecollaborants, including persons, machines, devices, neural networks,robots and algorithms, and heterogeneous networked teams composedthereof, with neurosynaptic network connectivity enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, that includesviewing, curating, annotating and tagging, using at least one or moresources of multichannel, multiplexed heterogeneous streaming imagerydata, including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.
 18. The method of claim 6 adapted forpersonalized precision targeted theranostic nanomedicine between andamong participant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, with neurosynaptic networkconnectivity enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instruction, aswell as recursive cognitive enrichment and collaborative knowledgeexchange, that includes viewing, curating, annotating and tagging, usingat least one or more sources of multichannel, multiplexed heterogeneousstreaming imagery data, including both medical and non-medical streamingimagery data, and combinations thereof, and together with images, video,modality imagery, waveforms, audio and haptic files,multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs.
 19. The method of claim 6adapted for managing clinical knowledge with networked metadatarepositories, including semantic metatdata repositories, between andamong participant cognitive collaborants, including persons, machines,devices, neural networks, robots and algorithms, and heterogeneousnetworked teams composed thereof, with neurosynaptic networkconnectivity enabling both synchronous and asynchronous multimodalclinical communications, collaboration, consultation and instruction, aswell as recursive cognitive enrichment and collaborative knowledgeexchange, that includes viewing, curating, annotating and tagging, usingat least one or more sources of multichannel, multiplexed heterogeneousstreaming imagery data, including both medical and non-medical streamingimagery data, and combinations thereof, and together with images, video,modality imagery, waveforms, audio and haptic files,multiomic—phenotypic, genomic, metabolomic, pathomic, radiomic,radiopathomic and radiogenomic—maps and clinical data sets, biometricmaps and movies, hapmaps, heat maps, data stream visualizations,structured reports, interactive media reports, clinical documents andkey performance indicators during various stages of medical disease andinjury management, including detection, diagnosis, prognosis, treatment,measurement, monitoring and reporting, as well as workflow optimizationwith operational analytics for outcomes, performance, results, resourceutilization, resource consumption and costs.
 20. The method of claim 6adapted for cognitive engineering with networked prediction machines,including augmented mind-machine decision making and augmented analyticswith network-connected edge devices, between and among participantcognitive collaborants, including persons, machines, devices, neuralnetworks, robots and algorithms, and heterogeneous networked teamscomposed thereof, with neurosynaptic network connectivity enabling bothsynchronous and asynchronous multimodal clinical communications,collaboration, consultation and instruction, as well as recursivecognitive enrichment and collaborative knowledge exchange, that includesviewing, curating, annotating and tagging, using at least one or moresources of multichannel, multiplexed heterogeneous streaming imagerydata, including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.
 21. A method enabling multichannel multiplexedcommunications, collaboration, consultation and instruction, as well asrecursive cognitive enrichment and collaborative knowledge exchange,with streaming imagery data by participant cognitive collaborants,including persons, machines, devices, neural networks, robots andalgorithms, and heterogeneous networked teams composed thereof, withmodular and scalable clusters of gateway streamer servers configured tosupport multiple network topologies, including peer-to-peer,hub-and-spoke, mesh chord and core-spine-leaf networks, as well as in2-tier, 3-tier, or N-tier application architectures, and heterogeneousnetwork combinations thereof, each gateway streamer server enablingneurosynaptic network connectivity enabling both synchronous andasynchronous multimodal clinical communications, collaboration,consultation and instruction, as well as recursive cognitive enrichmentand collaborative knowledge exchange, that includes viewing, curating,annotating and tagging, using at least one or more sources ofmultichannel, multiplexed heterogeneous streaming imagery data,including both medical and non-medical streaming imagery data, andcombinations thereof, and together with images, video, modality imagery,waveforms, audio and haptic files, multiomic—phenotypic, genomic,metabolomic, pathomic, radiomic, radiopathomic and radiogenomic—maps andclinical data sets, biometric maps and movies, hapmaps, heat maps, datastream visualizations, structured reports, interactive media reports,clinical documents and key performance indicators during various stagesof medical disease and injury management, including detection,diagnosis, prognosis, treatment, measurement, monitoring and reporting,as well as workflow optimization with operational analytics foroutcomes, performance, results, resource utilization, resourceconsumption and costs.